{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "corresponding-society",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "time: 466 µs (started: 2021-08-05 11:14:50 +08:00)\n"
     ]
    }
   ],
   "source": [
    "# 自动计算cell的计算时间\n",
    "%load_ext autotime"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "religious-domain",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash\n",
    "\n",
    "# 增加更新\n",
    "git add *.ipynb\n",
    "\n",
    "git remote -v\n",
    "\n",
    "git commit -m '更新 ch11 #2 change Aug 05, 2021'\n",
    "\n",
    "git push origin master"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "british-assessment",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "time: 6.28 s (started: 2021-08-05 11:15:01 +08:00)\n"
     ]
    }
   ],
   "source": [
    "#设置使用的gpu\n",
    "import tensorflow as tf\n",
    "\n",
    "gpus = tf.config.list_physical_devices(\"GPU\")\n",
    "\n",
    "if gpus:\n",
    "   \n",
    "    gpu0 = gpus[1] #如果有多个GPU，仅使用第0个GPU\n",
    "    tf.config.experimental.set_memory_growth(gpu0, True) #设置GPU显存用量按需使用\n",
    "    # 或者也可以设置GPU显存为固定使用量(例如：4G)\n",
    "    #tf.config.experimental.set_virtual_device_configuration(gpu0,\n",
    "    #    [tf.config.experimental.VirtualDeviceConfiguration(memory_limit=4096)]) \n",
    "    tf.config.set_visible_devices([gpu0],\"GPU\") "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "level-trauma",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "time: 22.2 ms (started: 2021-08-05 11:15:07 +08:00)\n"
     ]
    }
   ],
   "source": [
    "%config InlineBackend.figure_format='svg' #矢量图设置，让绘图更清晰"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "applied-capitol",
   "metadata": {},
   "source": [
    "# 文本深度学习"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "touched-profile",
   "metadata": {},
   "source": [
    "> **本章包含**\n",
    "> * 为机器学习应用程序预处理文本数据\n",
    "> * 用于文本处理的词袋方法和序列建模方法\n",
    "> * 变压器架构\n",
    "> * 序列到序列学习"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dominant-pocket",
   "metadata": {},
   "source": [
    "## 自然语言处理：鸟瞰图"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "collectible-wednesday",
   "metadata": {},
   "source": [
    "在计算机科学中，我们将人类语言（如英语或普通话）称为“自然”语言，以区别于专为机器设计的语言，如汇编、LISP 或 XML。每一种机器语言都是：它的起点是人类工程师设计的写下一组正式规则来描述你可以用这种语言做出什么陈述，以及它们的含义。规则首先出现，人们只有在规则集完成后才开始使用该语言。对于人类语言，情况正好相反：使用在先，规则在后。自然语言是由进化过程塑造的，就像生物有机体一样——这就是它“自然”的原因。它的“规则”，就像英语的语法一样，事后正式化，经常被用户忽视或破坏。因此，虽然机器可读语言是高度结构化和严谨的，使用精确的句法规则从固定的词汇表中将精确定义的概念编织在一起，但自然语言是混乱的——模棱两可、混乱、庞大且不断变化。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "mineral-realtor",
   "metadata": {},
   "source": [
    "创建能够理解自然语言的算法是一件大事：语言，尤其是文本，是我们大部分交流和大部分文化生产的基础。互联网主要是文本。语言是我们存储几乎所有知识的方式。我们的思想很大程度上建立在语言上。然而，理解自然语言的能力长期以来一直是机器所不具备的。有些人曾经天真地认为你可以简单地写下“英语规则集”，就像可以写下LISP的规则集一样。因此，构建自然语言处理 (NLP) 系统的早期尝试是通过“应用语言学”的视角进行的。工程师和语言学家会手工制作复杂的规则集来执行基本的机器翻译或创建简单的聊天机器人——比如 1960 年代著名的 ELIZA 程序，它使用模式匹配来维持非常基本的对话。但是语言是一种叛逆的东西，它不容易被形式化。经过几十年的努力，这些系统的能力仍然令人失望。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "operational-lancaster",
   "metadata": {},
   "source": [
    "手工制定的规则一直是 1990 年代的主导方法。但从 1980 年代后期开始，更快的计算机和更高的数据可用性开始使更好的替代方案变得可行。当您发现自己构建的系统包含大量临时规则时，作为一名聪明的工程师，您可能会开始问：我可以使用数据语料库来自动化查找这些规则的过程吗？我可以在某种规则空间中搜索规则，而不必自己想出来吗？就这样，你已经毕业从事机器学习。所以在 1980 年代后期，我们开始看到机器学习方法用于自然语言处理。最早的那些是基于决策树的——其意图实际上是使以前系统的那种 if/then/else 规则的开发自动化。然后统计方法开始加速，从逻辑回归开始。随着时间的推移，学习的参数模型完全接管了，语言学开始被视为更多的障碍而不是有用的工具。早期的语音识别研究员 Frederick Jelinek 在 1990 年代开玩笑说：“每次我解雇一名语言学家，语音识别器的性能都会上升”。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "technological-announcement",
   "metadata": {},
   "source": [
    "这就是现代 NLP 的意义所在：使用机器学习和大型数据集赋予计算机能力——不是语言，这是一个更崇高的目标——而是摄取一段理解语言作为输入并返回一些有用的东西，比如预测：\n",
    "* “这篇课文的主题是什么？” （文字分类）\n",
    "* “这段文字是否包含辱骂？” （内容过滤）\n",
    "* “这段文字听起来是正面的还是负面的？” （情绪分析）\n",
    "* “这不完整的句子中的下一个单词应该是什么？” （语言建模）\n",
    "* “这个用德语怎么说？” （翻译）\n",
    "* “你如何用一段话概括这篇文章？” （总结）\n",
    "* 等等。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "digital-aviation",
   "metadata": {},
   "source": [
    "当然，在本章中请记住，您将训练的文本处理模型并不具备人类对语言的理解； 相反，他们只是在输入数据中寻找统计规律——结果证明这足以在许多简单的任务上表现良好。 就像计算机视觉是应用于像素的模式识别一样，NLP 是应用于单词、句子和段落的模式识别。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "growing-memorial",
   "metadata": {},
   "source": [
    "NLP 的工具集——决策树、逻辑回归——从 1990 年代到 2010 年代初期才经历了缓慢的演变。 大多数研究重点是特征工程。 当我在 2013 年在 Kaggle 上赢得我的第一场 NLP 比赛时，我的模型，你猜对了，基于决策树和逻辑回归。 然而，在 2014-2015 年左右，事情终于开始发生变化。 多位研究人员开始研究循环神经网络的语言理解能力，特别是 LSTM——一种 1990 年代后期的序列处理算法，直到那时才受到关注"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "common-fishing",
   "metadata": {},
   "source": [
    "2015 年初，Keras 推出了第一个开源、易于使用的 LSTM 实现，这恰好是对循环神经网络重新产生兴趣的浪潮的开始——直到那时，只有“研究代码” 不能轻易重复使用。 然后从 2015 年到 2017 年，循环神经网络主导了蓬勃发展的 NLP 场景。 特别是双向 LSTM 模型在许多重要任务上设定了最新技术，从摘要到问答再到机器翻译。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "tight-society",
   "metadata": {},
   "source": [
    "最后，在 2017-2018 年左右，出现了一种取代 RNN 的新架构：Transformer，您将在本章的后半部分了解它。 Transformers 在短时间内在整个领域取得了长足的进步，如今大多数 NLP 系统都是基于它们的。\n",
    "\n",
    "让我们深入了解细节。 本章将带您从最基础的知识到使用 Transformer 进行机器翻译。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "communist-athletics",
   "metadata": {},
   "source": [
    "## 准备文本数据"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "serial-significance",
   "metadata": {},
   "source": [
    "作为可微函数的深度学习模型只能处理数字张量：它们不能将原始文本作为输入。 text 是将文本转换为数字向量化张量的过程。 文本矢量化过程有多种形状和形式，但它们都遵循相同的模板：\n",
    "* 首先，您将文本标准化以使其更易于处理，例如将其转换为小写或删除标点符号。\n",
    "* 您将文本拆分为多个单元（称为“标记”），例如字符、单词或单词组。 这称为标记化。\n",
    "* 您将每个这样的标记转换为数字向量。 这通常涉及首先索引数据中存在的所有标记。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ecological-hobby",
   "metadata": {},
   "source": [
    "![](https://tva1.sinaimg.cn/large/008i3skNgy1gt1np624r0j310a0s2jte.jpg)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "organic-remainder",
   "metadata": {},
   "source": [
    "让我们回顾一下每个步骤。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "atmospheric-contributor",
   "metadata": {},
   "source": [
    "### 文本标准化"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cubic-bacon",
   "metadata": {},
   "source": [
    "考虑这两个句子：\n",
    "* “日落来了。我盯着墨西哥的天空。大自然是不是很壮观？？”\n",
    "* “日落来了，我凝视着墨西哥的天空。大自然是不是很精彩？”"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "psychological-proceeding",
   "metadata": {},
   "source": [
    "它们非常相似——事实上，它们几乎完全相同。 然而，如果您将它们转换为字节字符串，它们最终会得到非常不同的表示，因为“i”和“I”是两个不同的字符，“Mexico”和“México”是两个不同的词，“isnt”不是 't “不是”，等等。 机器学习模型不知道“i”和“I”是同一个字母，“é”是带重音的“e”，或者“凝视”和“凝视”是两种形式的 同一个动词。\n",
    "\n",
    "文本标准化是特征工程的一种基本形式，旨在消除您不希望模型必须处理的编码差异。 它也不是机器学习独有的——如果你正在构建一个搜索引擎，你必须做同样的事情。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "level-potato",
   "metadata": {},
   "source": [
    "最简单和最广泛的标准化方案之一是“转换为小写并删除标点符号”。 我们的两句话会变成：\n",
    "* “日落来了，我凝视着墨西哥的天空，这不是大自然的壮丽景色”\n",
    "* “日落来了，我凝视着墨西哥的天空，大自然并不灿烂”"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "possible-england",
   "metadata": {},
   "source": [
    "已经很近了。 另一种常见的转换是将特殊字符转换为标准形式，例如将“é”替换为“e”，将“æ”替换为“ae”等。 我们的代币“méxico”将变成“mexico”。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "sufficient-reward",
   "metadata": {},
   "source": [
    "最后，在机器学习上下文中很少使用的更高级的标准化模式是词干：将术语的变体（例如动词的不同共轭词干形式）转换为单个共享表示，例如将“捕获”转换为“捕获” 和“一直在捕捉”变成“[catch]”或“cats”变成“[cat]”。 通过词干提取，“was staring”和“stared”会变成类似于“[stare]”的东西——我们两个相似的句子最终会以相同的编码结束：\n",
    "\n",
    "* “日落来了我[凝视]墨西哥的天空不是大自然的壮丽”"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "responsible-homework",
   "metadata": {},
   "source": [
    "使用这些标准化技术，您的模型将需要更少的训练数据并且可以更好地泛化——它不需要“日落”和“日落”的大量示例来了解它们的含义相同，并且它能够有意义 “México”，即使它在训练集中只看到“mexico”。 当然，标准化也可能会擦除一些信息，因此请始终牢记上下文：例如，如果您正在编写一个从采访文章中提取问题的模型，它肯定应该处理“？” 作为一个单独的令牌而不是丢弃它，因为它是这个特定任务的有用信号。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "obvious-allah",
   "metadata": {},
   "source": [
    "### 文本分割（标记化）"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "tired-hormone",
   "metadata": {},
   "source": [
    "文本标准化后，您需要将其分解为要向量化的单元（标记），这一步骤称为“标记化”。 您可以通过三种不同的方式执行此操作：\n",
    "* 单词级标记化，其中标记是空格分隔（或标点分隔）的子字符串。 一种变体是在适用时进一步将单词拆分为子词，例如将“凝视”视为“star+ing”或“call”视为“call+ed”。\n",
    "* N-gram 标记化，其中标记是 N 个连续单词的组。 例如，“the cat”或“he was”将是 2-gram 标记（也称为 bigrams）。\n",
    "* 字符级标记化，其中每个字符都是它自己的标记。 在实践中，这种方案很少使用，你只能在专门的上下文中看到它，比如文本生成或语音识别。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "catholic-huntington",
   "metadata": {},
   "source": [
    "通常，您将始终使用词级或 N-gram 标记化。 有两种文本处理模型：一种关心词序，称为“序列模型”，另一种将输入词视为一个集合，丢弃其原始顺序，称为“词袋模型”。 如果您正在构建一个序列模型，您将使用词级标记化，如果您正在构建一个词袋模型，您将使用 N-gram 标记化：N-grams 是一种人工注入的方法 少量的局部词序信息进入模型。 在本章中，您将详细了解每种类型的模型以及何时使用它们。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "irish-powder",
   "metadata": {},
   "source": [
    "> **理解 N-gram 和词袋**\n",
    ">\n",
    "> 单词 N-gram 是可以从句子中提取的 N 个（或更少）连续单词的组。 相同的概念也可以应用于字符而不是单词。\n",
    "> \n",
    "> 这是一个简单的例子。 考虑句子“the cat sat on the mat”。 它可以分解为以下一组 2-gram：\n",
    "> ```\n",
    "> {\"the\", \"the cat\", \"cat\", \"cat sat\", \"sat\",\n",
    ">  \"sat on\", \"on\", \"on the\", \"the mat\", \"mat\"}\n",
    "> ```\n",
    "> \n",
    "> 它也可以分解为以下一组 3-gram：\n",
    "> ```\n",
    "> {\"the\", \"the cat\", \"cat\", \"cat sat\", \"the cat sat\",\n",
    ">  \"sat\", \"sat on\", \"on\", \"cat sat on\", \"on the\",\n",
    ">  \"sat on the\", \"the mat\", \"mat\", \"on the mat\"}\n",
    "> ```\n",
    "> \n",
    "> 这样的集合分别称为 bag-of-2-grams bag-of-3-grams 或 。这里的术语包指的是您正在处理一组标记而不是列表或序列：标记没有特定的顺序。这一系列标记化方法称为词袋袋N-grams（或）。\n",
    "> \n",
    "> 因为bag-of-words不是保序的tokenization方法（生成的token被理解为一个集合，而不是一个序列，丢失了句子的一般结构），> 所以它倾向于用于浅层语言处理模型而不是深度学习模型。提取 N-gram 是特征工程的一种形式，深度学习序列模型摒弃了这种手动方法，取而代之的是分层特征学习。通过查看连续的单词或字符序列，一维 convnets、循环神经网络和 Transformer 能够学习单词和字符组的表示，而无需明确告知此类组的存在。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "invisible-romantic",
   "metadata": {},
   "source": [
    "### 词汇索引"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "sound-contamination",
   "metadata": {},
   "source": [
    "将文本拆分为标记后，您需要将每个标记编码为数字表示。 您可能会以无状态的方式执行此操作，例如，通过将每个标记散列到一个固定的二进制向量中——但实际上，您要做的方法是构建训练数据中找到的所有术语的索引（ “词汇表”），并为词汇表中的每个条目分配一个唯一的整数。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "weird-assumption",
   "metadata": {},
   "source": [
    "像这样的东西："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "virgin-passenger",
   "metadata": {},
   "outputs": [],
   "source": [
    "vocabulary = {}\n",
    "for text in dataset:\n",
    "    text = standardize(text)\n",
    "    tokens = tokenize(text)\n",
    "    for token in tokens:\n",
    "        if token not in vocabulary:\n",
    "            vocabulary[token] = len(vocabulary)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "sacred-reynolds",
   "metadata": {},
   "source": [
    "然后，您可以将该整数转换为可由神经网络处理的向量编码，例如单热向量："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "standing-coverage",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "time: 746 µs (started: 2021-08-01 22:55:45 +08:00)\n"
     ]
    }
   ],
   "source": [
    "def one_hot_encode_token(token):\n",
    "    vector = np.zeros((len(vocabulary),))\n",
    "    token_index = vocabulary[token]\n",
    "    vector[token_index] = 1\n",
    "    return vector"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fantastic-valley",
   "metadata": {},
   "source": [
    "请注意，在此步骤中，通常会将词汇表限制为仅在训练数据中找到的前 20,000 个或 30,000 个最常见的单词。 任何文本数据集都倾向于包含大量独特的术语，其中大部分只出现一两次——索引这些稀有术语会导致特征空间过大，其中大多数特征几乎没有信息内容。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "varying-ballot",
   "metadata": {},
   "source": [
    "还记得你在第 4 章和第 5 章中在 IMDB 数据集上训练你的第一个深度学习模型吗？ 您使用的数据来自 keras.datasets.imdb，已经被预处理成整数序列，其中每个整数代表一个给定的单词。 当时，我们使用设置 num_words=10000，以将我们的词汇量限制在训练数据中最常见的前 10,000 个单词。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "secure-rating",
   "metadata": {},
   "source": [
    "现在，这里有一个我们不应忽视的重要细节：当我们在词汇索引中查找新标记时，它可能不一定存在。 您的训练数据可能不包含“cherimoya”一词的任何实例（或者您可能因为它太罕见而将其从索引中排除），因此执行 token_index =vocabulary[\"cherimoya\"] 可能会导致 KeyError。 为了解决这个问题，您应该使用“词汇外”索引（缩写为 OOV 索引），它是所有不在索引中的标记的统称。 它通常是索引 1：你实际上是在做 token_index =vocabulary.get(token, 1)。 将整数序列解码回单词时，您将 1 替换为“[UNK]”（您将其称为“OOV 标记”）之类的内容。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "covered-reasoning",
   "metadata": {},
   "source": [
    "“为什么使用 1 而不是 0？”，您可能会问。 那是因为 0 已经被占用了。 您通常会使用两种特殊标记：OOV 标记（索引 1）和“掩码标记”（索引 0）。 虽然 OOV 令牌的意思是“这是一个我们不认识的词”，但掩码令牌告诉我们“忽略我，我不是一个词”。 你会特别用它来填充序列数据：因为数据批次需要是连续的，一批序列数据中的所有序列必须具有相同的长度，所以较短的序列应该填充到最长序列的长度。 如果你想用序列 [5, 7, 124, 4, 89] 和 [8, 34, 21] 制作一批数据，它必须是这样的：\n",
    "```\n",
    "[[5, 7, 124, 4, 89]\n",
    " [8, 34, 21, 0, 0]]\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "stuffed-belgium",
   "metadata": {},
   "source": [
    "您在第 4 章和第 5 章中使用过的 IMDB 数据集整数序列批次以这种方式填充了零。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "serial-flesh",
   "metadata": {},
   "source": [
    "### 使用 `TextVectorization` 层"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "stunning-premises",
   "metadata": {},
   "source": [
    "到目前为止，我们介绍的每一步都可以很容易地在纯 Python 中实现。 也许你可以这样写："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "central-weather",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "time: 1.66 ms (started: 2021-08-01 23:15:27 +08:00)\n"
     ]
    }
   ],
   "source": [
    "import string\n",
    "\n",
    "class Vectorizer:\n",
    "    def standardize(self, text):\n",
    "        text = text.lower()\n",
    "        return \"\".join(char for char in text if char not in string.punctuation)\n",
    "    \n",
    "    def tokenize(self, text):\n",
    "        text = self.standardize(text)\n",
    "        return text.split()\n",
    "    \n",
    "    def make_vocabulary(self, dataset):\n",
    "        self.vocabulary = {\"\": 0, \"[UNK]\": 1}\n",
    "        for text in dataset:\n",
    "            text = self.standardize(text)\n",
    "            tokens = self.tokenize(text)\n",
    "            for token in tokens:\n",
    "                if token not in self.vocabulary:\n",
    "                    self.vocabulary[token] = len(self.vocabulary)\n",
    "        self.inverse_vocabulary = dict((v, k) for k, v in self.vocabulary.items())\n",
    "\n",
    "    def encode(self, text):\n",
    "        text = self.standardize(text)\n",
    "        tokens = self.tokenize(text)\n",
    "        return [self.vocabulary.get(token, 1) for token in tokens]\n",
    "    \n",
    "    def decode(self, int_sequence):\n",
    "        return \" \".join(self.inverse_vocabulary.get(i, \"[UNK]\") for i in int_sequence)\n",
    "    \n",
    "vectorizer = Vectorizer()\n",
    "\n",
    "dataset = [\"I write, erase, rewrite\",\n",
    "           \"Erase again, and then\",\n",
    "           \"A poppy blooms.\",\n",
    "          ]\n",
    "\n",
    "vectorizer.make_vocabulary(dataset)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "complimentary-strength",
   "metadata": {},
   "source": [
    "它可以完成以下工作："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "voluntary-shark",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[2, 3, 5, 7, 1, 5, 6]"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "time: 13.2 ms (started: 2021-08-01 23:16:11 +08:00)\n"
     ]
    }
   ],
   "source": [
    "test_sentence = \"I write, rewrite, and still rewrite again\"\n",
    "encoded_sentence = vectorizer.encode(test_sentence)\n",
    "encoded_sentence"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "disabled-minority",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'i write rewrite and [UNK] rewrite again'"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "time: 4.49 ms (started: 2021-08-01 23:18:50 +08:00)\n"
     ]
    }
   ],
   "source": [
    "decoded_sentence = vectorizer.decode(encoded_sentence)\n",
    "decoded_sentence"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "distinct-contamination",
   "metadata": {},
   "source": [
    "然而，使用这样的东西不会有很好的性能。 在实践中，您将使用 Keras TextVectorization 层，它快速高效，可以直接放入 tf.data 管道或 Keras 模型中。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "widespread-eligibility",
   "metadata": {},
   "source": [
    "这是 TextVectorization 层的样子："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "crude-enough",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "time: 21.8 ms (started: 2021-08-03 08:18:56 +08:00)\n"
     ]
    }
   ],
   "source": [
    "from tensorflow.keras.layers.experimental.preprocessing import TextVectorization\n",
    "\n",
    "# 配置层以返回编码为整数索引的单词序列。 还有其他几种可用的输出模式，我们将在稍后看到。\n",
    "text_vectorization = TextVectorization( output_mode=\"int\", )"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "incredible-cosmetic",
   "metadata": {},
   "source": [
    "默认情况下，TextVectorization 层将使用设置“转换为小写并删除标点符号”进行文本标准化，并使用“拆分空白”进行标记化。 但重要的是，您可以为标准化和标记化提供自定义函数，这意味着该层足够灵活以处理任何用例。 请注意，此类自定义函数应该对张量进行操作，而不是常规 Python 字符串！ 例如，默认层 tf.string 行为等效于以下内容："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "characteristic-highland",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "time: 18.6 ms (started: 2021-08-01 23:31:05 +08:00)\n"
     ]
    }
   ],
   "source": [
    "import re\n",
    "import string\n",
    "import tensorflow as tf\n",
    "\n",
    "def custom_standardization_fn(string_tensor):\n",
    "#     将字符串转换为小写\n",
    "    lowercase_string = tf.strings.lower(string_tensor) \n",
    "#     用空字符串替换标点符号\n",
    "    return tf.strings.regex_replace(\n",
    "        lowercase_string, f\"[{re.escape(string.punctuation)}]\", \"\")\n",
    "\n",
    "def custom_split_fn(string_tensor):\n",
    "#     在空白处拆分字符串\n",
    "    return tf.strings.split(string_tensor)\n",
    "\n",
    "text_vectorization = TextVectorization(output_mode=\"int\",\n",
    "                                       standardize=custom_standardization_fn,\n",
    "                                       split=custom_split_fn,)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "eastern-protest",
   "metadata": {},
   "source": [
    "要索引文本语料库的词汇表，只需使用生成字符串的 Dataset 对象或仅使用 Python 字符串列表调用图层的 adapt() 方法："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "retained-gates",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "time: 1.17 s (started: 2021-08-01 23:33:29 +08:00)\n"
     ]
    }
   ],
   "source": [
    "dataset = [\n",
    "    \"I write, erase, rewrite\",\n",
    "    \"Erase again, and then\",\n",
    "    \"A poppy blooms.\",]\n",
    "\n",
    "text_vectorization.adapt(dataset)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "corresponding-tender",
   "metadata": {},
   "source": [
    "请注意，您可以通过 get_vocabulary() 检索计算出的词汇表——如果您需要将编码为整数序列的文本转换回单词，这会很有用。 词汇表中的前两个条目是掩码标记（索引 0）和 OOV 标记（索引 1）。 词汇表中的条目按频率排序——因此对于真实世界的数据集，诸如“the”或“a”之类的非常常见的词将排在第一位。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "jewish-hydrogen",
   "metadata": {},
   "source": [
    "> 清单 11.1 显示词汇表"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "technical-policy",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['',\n",
       " '[UNK]',\n",
       " 'erase',\n",
       " 'write',\n",
       " 'then',\n",
       " 'rewrite',\n",
       " 'poppy',\n",
       " 'i',\n",
       " 'blooms',\n",
       " 'and',\n",
       " 'again',\n",
       " 'a']"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "time: 22.5 ms (started: 2021-08-01 23:35:14 +08:00)\n"
     ]
    }
   ],
   "source": [
    "text_vectorization.get_vocabulary()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "hindu-genetics",
   "metadata": {},
   "source": [
    "为了演示，让我们尝试编码然后解码一个例句："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "satisfactory-offset",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "time: 2.65 ms (started: 2021-08-01 23:36:19 +08:00)\n"
     ]
    }
   ],
   "source": [
    "vocabulary = text_vectorization.get_vocabulary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "split-billion",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "time: 734 µs (started: 2021-08-01 23:36:28 +08:00)\n"
     ]
    }
   ],
   "source": [
    "test_sentence = \"I write, rewrite, and still rewrite again\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "excess-shuttle",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(7,), dtype=int64, numpy=array([ 7,  3,  5,  9,  1,  5, 10])>"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "time: 59.3 ms (started: 2021-08-01 23:36:58 +08:00)\n"
     ]
    }
   ],
   "source": [
    "encoded_sentence = text_vectorization(test_sentence)\n",
    "encoded_sentence"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "egyptian-thunder",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{0: '',\n",
       " 1: '[UNK]',\n",
       " 2: 'erase',\n",
       " 3: 'write',\n",
       " 4: 'then',\n",
       " 5: 'rewrite',\n",
       " 6: 'poppy',\n",
       " 7: 'i',\n",
       " 8: 'blooms',\n",
       " 9: 'and',\n",
       " 10: 'again',\n",
       " 11: 'a'}"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "time: 6.61 ms (started: 2021-08-01 23:37:26 +08:00)\n"
     ]
    }
   ],
   "source": [
    "inverse_vocab = dict(enumerate(vocabulary))\n",
    "inverse_vocab"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "chief-interface",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'i write rewrite and [UNK] rewrite again'"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "time: 52.8 ms (started: 2021-08-01 23:37:53 +08:00)\n"
     ]
    }
   ],
   "source": [
    "decoded_sentence = \" \".join(inverse_vocab[int(i)] for i in encoded_sentence)\n",
    "decoded_sentence  "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "civil-richardson",
   "metadata": {},
   "source": [
    "在 tf.data 管道中或作为模型的一部分使用 TextVectorization 层重要的是，因为 TextVectorization 主要是一个字典查找操作，它不能在 GPU（或 TPU）上执行——只能在 CPU 上执行。 因此，如果您在 GPU 上训练模型，那么 TextVectorization 层将在 CPU 上运行，然后将其输出发送到 GPU。 这具有重要的性能影响。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "regulated-hands",
   "metadata": {},
   "source": [
    "我们可以通过两种方式使用我们的 TextVectorization 层。 第一个选项是将其放入管道中，如下所示：tf.data"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "tracked-elevation",
   "metadata": {},
   "source": [
    "> 清单 11.2 选项 1：tf.data 管道中的 TextVectorization"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "dietary-waters",
   "metadata": {},
   "outputs": [],
   "source": [
    "int_sequence_dataset = string_dataset.map(text_vectorization)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "blank-township",
   "metadata": {},
   "source": [
    "第二种选择是让它成为模型的一部分（毕竟它是一个 Keras 层），就像这样："
   ]
  },
  {
   "cell_type": "markdown",
   "id": "wooden-valuable",
   "metadata": {},
   "source": [
    "> 清单 11.3 选项 2：模型中的 TextVectorization"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "blocked-coaching",
   "metadata": {},
   "outputs": [],
   "source": [
    "# 创建一个需要字符串的符号输入。\n",
    "text_input = keras.Input(shape=(), dtype=\"string\") \n",
    "\n",
    "# 对其应用文本矢量化层。\n",
    "vectorized_text = text_vectorization(text_input) \n",
    "\n",
    "# 您可以继续在顶部链接新层——只是您的常规功能 API 模型。\n",
    "embedded_input = keras.layers.Embedding(...)(vectorized_text) \n",
    "output = ... \n",
    "\n",
    "model = keras.Model(text_input, output)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "improved-logan",
   "metadata": {},
   "source": [
    "> 两者之间有一个重要区别：如果矢量化步骤是模型的一部分，它将与模型的其余部分同步发生。 这意味着在每个训练步骤中，模型的其余部分（放置在 GPU 上）必须等待 TextVectorization 层（放置在 CPU 上）的输出准备就绪才能开始工作（见图 TODO） . 同时，将该层放入 tf.data 管道中，您可以在 CPU 上对数据进行异步预处理：当 GPU 在一批矢量化数据上运行模型时，CPU 通过矢量化下一批原始字符串来保持忙碌。\n",
    "> \n",
    "> 因此，如果您在 GPU 或 TPU 上训练模型，您可能希望使用选项 1 以获得最佳性能。 这就是我们将在本章中的所有实际示例中执行的操作。 但是，在 CPU 上进行训练时，同步处理很好：无论您选择选项 1 还是选项 2，您都将获得 100% 的内核利用率。\n",
    ">\n",
    "> 现在，如果您要将我们的模型导出到生产环境，您可能希望发布一个接受原始字符串作为输入的模型，就像上面选项二的代码片段一样——否则，您将不得不重新实现文本标准化和标记化 您的生产环境（可能在 JavaScript 中？），并且您将面临引入小的预处理差异的风险，这会损害模型的准确性。 值得庆幸的是，TextVectorization 层使您能够将文本预处理直接包含到您的模型中，从而使其更易于部署——即使您最初将该层用作管道的一部分。 在 TODO 部分，您将学习如何将 tf.data 导出为包含文本预处理的仅推理训练模型。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "surrounded-enough",
   "metadata": {},
   "source": [
    "到目前为止，您已经了解了有关文本预处理的所有知识——让我们进入建模阶段。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "buried-material",
   "metadata": {},
   "source": [
    "## 表示词组的两种方法：集合和序列"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "rural-uzbekistan",
   "metadata": {},
   "source": [
    "机器学习模型应该如何表示单个单词是一个相对没有争议的问题：它们是分类特征（来自预定义集合的值），我们知道如何处理这些特征。 它们应该被编码为特征空间中的维度，或作为类别向量（在这种情况下是词向量）。 然而，一个更成问题的问题是如何对单词编织成句子的方式进行编码：词序。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "compliant-textbook",
   "metadata": {},
   "source": [
    "自然语言中的顺序问题是一个有趣的问题：与时间序列的步骤不同，句子中的单词没有自然的规范顺序。 不同的语言以非常不同的方式对相似的单词进行排序。 例如，英语的句子结构与日语的句子结构大不相同。 即使在给定的语言中，您通常也可以通过稍微重新排列单词以不同的方式说出相同的内容。 更进一步，如果你将一个短句中的单词完全随机化，你仍然可以在很大程度上弄清楚它在说什么——尽管在许多情况下似乎会出现明显的歧义。 秩序显然很重要，但它与意义的关系并不简单。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "patent-square",
   "metadata": {},
   "source": [
    "如何表示词序是不同类型 NLP 架构产生的关键问题。 您可以做的最简单的事情就是丢弃顺序并将文本视为一组无序的单词——这为您提供了词袋模型。 您还可以决定严格按照单词出现的顺序处理单词，一次一个，就像时间序列中的步骤一样——然后您可以利用上一章中的循环模型。 最后，混合方法也是可能的：Transformer 架构在技术上与顺序无关，但它将词位置信息注入它处理的表示中，这使其能够同时查看句子的不同部分（与 RNN 不同），同时仍然 订单意识。 因为它们考虑了词序，所以 RNN 和 Transformer 都被称为序列模型。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "seventh-figure",
   "metadata": {},
   "source": [
    "从历史上看，机器学习在 NLP 中的大多数早期应用只涉及词袋模型。 随着循环神经网络的重生，对序列模型的兴趣在 2015 年才开始上升。 今天，这两种方法仍然适用。 让我们看看它们是如何工作的，以及何时利用它们。\n",
    "\n",
    "我们将在一个著名的文本分类基准上演示每种方法：IMDB 电影评论情感分类数据集。 在第 4 章和第 5 章中，您已经使用了 IMDB 数据集的预矢量化版本——现在，让我们处理原始 IMDB 文本数据，就像您在现实世界中处理新的文本分类问题时所做的那样。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "opposed-ground",
   "metadata": {},
   "source": [
    "### 准备 IMDB 电影评论数据"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "legal-investigator",
   "metadata": {},
   "source": [
    "让我们从 Andrew Maas 的斯坦福页面下载数据集开始，并解压缩它："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "portuguese-introduction",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current\n",
      "                                 Dload  Upload   Total   Spent    Left  Speed\n",
      "100 80.2M  100 80.2M    0     0   315k      0  0:04:20  0:04:20 --:--:--  321k\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "time: 6min 31s (started: 2021-08-01 23:53:57 +08:00)\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "\n",
    "cd data\n",
    "\n",
    "curl -O https://ai.stanford.edu/~amaas/data/sentiment/aclImdb_v1.tar.gz && tar -xf aclImdb_v1.tar.gz"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "unlikely-piano",
   "metadata": {},
   "source": [
    "剩下一个名为 aclImdb 的目录，其结构如下：\n",
    "```\n",
    "aclImdb/\n",
    "...train/\n",
    "......pos/\n",
    "......neg/\n",
    "...test/\n",
    "......pos/\n",
    "......neg/\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "together-chassis",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "time: 99.9 ms (started: 2021-08-02 00:06:29 +08:00)\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "\n",
    "cd data/aclImdb/"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "alpine-ireland",
   "metadata": {},
   "source": [
    "例如，train/pos/ 目录包含一组 12,500 个文本文件，每个文件都包含用作训练数据的正面情感电影评论的文本正文。 负面情绪评论存在于neg目录中。 总共有 25,000 个文本文件用于负训练，另外 25,000 个用于测试。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "floating-sensitivity",
   "metadata": {},
   "source": [
    "那里还有一个 train/unsup 子目录，我们不需要它。 让我们删除它："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "heard-stationery",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "time: 236 ms (started: 2021-08-02 00:07:50 +08:00)\n"
     ]
    }
   ],
   "source": [
    "![ -d data/aclImdb/train/unsup ] && rm -r data/aclImdb/train/unsup"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "representative-jefferson",
   "metadata": {},
   "source": [
    "看看其中一些文本文件的内容。 无论您是处理文本数据还是图像数据，请记住在深入建模之前始终检查数据的外观。 它将为您的模型实际做什么奠定您的直觉。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "smoking-remains",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I first saw this back in the early 90s on UK TV, i did like it then but i missed the chance to tape it, many years passed but the film always stuck with me and i lost hope of seeing it TV again, the main thing that stuck with me was the end, the hole castle part really touched me, its easy to watch, has a great story, great music, the list goes on and on, its OK me saying how good it is but everyone will take there own best bits away with them once they have seen it, yes the animation is top notch and beautiful to watch, it does show its age in a very few parts but that has now become part of it beauty, i am so glad it has came out on DVD as it is one of my top 10 films of all time. Buy it or rent it just see it, best viewing is at night alone with drink and food in reach so you don't have to stop the film.<br /><br />Enjoytime: 184 ms (started: 2021-08-02 00:08:32 +08:00)\n"
     ]
    }
   ],
   "source": [
    "!cat data/aclImdb/train/pos/4077_10.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "suffering-title",
   "metadata": {},
   "source": [
    "接下来，让我们通过在新目录 aclImdb/val 中设置 20% 的训练文本文件来准备验证集："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 107,
   "id": "severe-passing",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "time: 3.24 s (started: 2021-08-02 00:58:44 +08:00)\n"
     ]
    }
   ],
   "source": [
    "import os, pathlib, shutil, random\n",
    "\n",
    "base_dir = pathlib.Path(\"data/aclImdb\")\n",
    "val_dir = base_dir/\"val\"\n",
    "train_dir = base_dir/\"train\"\n",
    "\n",
    "for category in (\"neg\", \"pos\"):\n",
    "    os.makedirs(val_dir / category, exist_ok=True)\n",
    "    files = os.listdir(train_dir / category)\n",
    "    random.Random(1337).shuffle(files)\n",
    "    num_val_samples = int(0.2 * len(files)) \n",
    "    val_files = files[-num_val_samples:]\n",
    "    for fname in val_files:\n",
    "        shutil.move(train_dir/category/fname, val_dir/category/fname)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "accepting-enlargement",
   "metadata": {},
   "source": [
    "还记得在第 8 章中，我们如何使用实用程序 image_dataset_from_directory 为目录结构创建一批图像及其标签吗？ 您可以使用实用程序 text_dataset_from_directory 对文本文件执行完全相同的数据集操作。 让我们创建三个 Dataset 对象，用于训练、验证和测试："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "separated-finland",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Found 20000 files belonging to 2 classes.\n",
      "Found 5000 files belonging to 2 classes.\n",
      "Found 25000 files belonging to 2 classes.\n",
      "time: 4.99 s (started: 2021-08-04 14:14:27 +08:00)\n"
     ]
    }
   ],
   "source": [
    "from tensorflow import keras\n",
    "\n",
    "batch_size = 32\n",
    "\n",
    "train_ds = keras.preprocessing.text_dataset_from_directory( \n",
    "    \"data/aclImdb/train\", batch_size=batch_size)\n",
    "\n",
    "val_ds = keras.preprocessing.text_dataset_from_directory(\n",
    "    \"data/aclImdb/val\", batch_size=batch_size)\n",
    "\n",
    "test_ds = keras.preprocessing.text_dataset_from_directory(\n",
    "    \"data/aclImdb/test\", batch_size=batch_size)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "planned-israel",
   "metadata": {},
   "source": [
    "运行这一行应该输出“Found 20000 files分为2个类”； 如果您看到“发现 70000 个文件属于 3 个类”，则表示您忘记删除 aclImdb/train/unsup 目录。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "demographic-canyon",
   "metadata": {},
   "source": [
    "这些数据集产生的输入是 TensorFlow 张量，目标是编码值“0”或“1”的 tf.string int32 张量："
   ]
  },
  {
   "cell_type": "markdown",
   "id": "sharing-asthma",
   "metadata": {},
   "source": [
    "> 清单 11.4 显示第一批的形状和数据类型"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "dental-milan",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "inputs.shape: (32,)\n",
      "inputs.dtype: <dtype: 'string'>\n",
      "targets.shape: (32,)\n",
      "targets.dtype: <dtype: 'int32'>\n",
      "inputs[0]: tf.Tensor(b'Ah, the best and funniest movie about female football fans, only slightly better than the 1982 saga of teenage delusion set in North London (qv). By the way, I just watched this on Film 4 [2008-12-21] and am ruing my inability to set the PVR).<br /><br />This is easily my second favourite football movie after \"Mike Bassett: England Manager\", but this time with the added twist of looking like a guerrilla piece of movie-making from a team who apparently keep making movies which are banned in the country in which they are made (just think about the bit where the girls are taken from the stadium just as the Sun is setting: fast reactions all round). <br /><br />It is rare for a movie to make me laugh out loud, but when the rural soldier escorted one of the girls into the lavvies while forcing her to wear an inpromptu mask made from a poster of Ali Daei, I couldn\\'t stop waking the others in the house with my snorts, especially when the young soldier misunderstood the grandfather\\'s calls for assistance... <br /><br />Speaking of the toilets, I just wish I could speak Farsi so that I could work out the writing on the toilet walls (yes, there were a few scribblings in Roman script, but they mostly referred to wanky American rock bands).<br /><br />Also, on the rural-and-Farsi theme, don\\'t you reckon that Omid Djalili sounds like a yokel when he talks Farsi? Listen to the custody sergeant in this movie and then go \"Yup.\"<br /><br />I\\'m not going to bore on about the sexual politics of Iran, and I\\'m not going to bore on about the dubious acting; my love for this movie comes from the script and the editing: top notch stuff.<br /><br />Here\\'s a list of my top favourite football movies: 1. Mike Bassett, England Manager 2. Offside 3. Those Glory Glory Days 4. A Shot at Glory 5. The Arsenal Stadium Mystery', shape=(), dtype=string)\n",
      "targets[0]: tf.Tensor(1, shape=(), dtype=int32)\n",
      "time: 110 ms (started: 2021-08-04 14:14:32 +08:00)\n"
     ]
    }
   ],
   "source": [
    "for inputs, targets in train_ds:\n",
    "    print(\"inputs.shape:\", inputs.shape)\n",
    "    print(\"inputs.dtype:\", inputs.dtype)\n",
    "    print(\"targets.shape:\", targets.shape)\n",
    "    print(\"targets.dtype:\", targets.dtype)\n",
    "    print(\"inputs[0]:\", inputs[0])\n",
    "    print(\"targets[0]:\", targets[0])\n",
    "    break"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ordinary-democracy",
   "metadata": {},
   "source": [
    "可以了，好了。 现在让我们尝试从这些数据中学习一些东西。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "legendary-whole",
   "metadata": {},
   "source": [
    "### 将单词作为集合处理：词袋方法"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "thorough-irish",
   "metadata": {},
   "source": [
    "对一段文本进行编码以供机器学习模型处理的最简单方法是丢弃顺序并将其视为一组（“袋子”）标记。 您可以查看单个单词（unigrams），也可以通过查看连续标记组（N-grams）来尝试恢复一些本地顺序信息。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "occasional-lounge",
   "metadata": {},
   "source": [
    "**二进制编码的单字（UNIGRAMS）**"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cheap-hours",
   "metadata": {},
   "source": [
    "如果用一袋单字，句子“the cat sat on the mat”就变成了：\n",
    "```Python\n",
    "{\"cat\", \"mat\", \"on\", \"sat\", \"the\"}\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "applied-independence",
   "metadata": {},
   "source": [
    "这种编码的主要优点是您可以将整个文本表示为单个向量，其中每个条目都是给定单词的存在指示符。 例如，使用二进制编码（multi-hot），您可以将文本编码为一个向量，其维度与词汇表中的单词数量一样多——0 几乎无处不在，一些 1 表示对文本中存在的单词进行编码的维度 . 这就是我们在第 4 章和第 5 章中处理文本数据时所做的。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "conventional-victorian",
   "metadata": {},
   "source": [
    "让我们在我们的任务中试试这个。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "tight-cambridge",
   "metadata": {},
   "source": [
    "首先，让我们用 TextVectorization 层准备处理我们的原始文本数据集，以便它们产生二进制词向量。 我们的层只会查看单个单词（也就是说，unigrams）。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "voluntary-manual",
   "metadata": {},
   "source": [
    "> 清单 11.5 使用 TextVectorization 层预处理我们的数据集"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "electrical-singapore",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "time: 5.71 s (started: 2021-08-04 14:17:58 +08:00)\n"
     ]
    }
   ],
   "source": [
    "from tensorflow.keras.layers.experimental.preprocessing import TextVectorization\n",
    "\n",
    "# 将词汇量限制为 20,000 个最常用的单词。 否则，我们将索引训练数据中的每个单词——可能有数万个只出现一次或两次的术语，\n",
    "# 因此没有信息量。 一般来说，20,000 是适合文本分类的词汇量。\n",
    "text_vectorization = TextVectorization(max_tokens=20000, \n",
    "                                       output_mode=\"binary\", # 将输出标记编码为二进制向量。\n",
    "                                      )\n",
    "\n",
    "# 准备一个仅产生原始文本输入（无标签）的数据集。\n",
    "text_only_train_ds = train_ds.map(lambda x, y: x) \n",
    "# 通过adapter() 方法，使用该数据集来索引数据集词汇表。\n",
    "text_vectorization.adapt(text_only_train_ds) \n",
    "\n",
    "# 准备我们的训练、验证和测试数据集的处理版本。\n",
    "binary_1gram_train_ds = train_ds.map(lambda x, y: (text_vectorization(x), y)) \n",
    "binary_1gram_val_ds = val_ds.map(lambda x, y: (text_vectorization(x), y))\n",
    "binary_1gram_test_ds = test_ds.map(lambda x, y: (text_vectorization(x), y))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "lined-speed",
   "metadata": {},
   "source": [
    "您可以尝试检查以下数据集之一的输出："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "id": "likely-tribute",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "inputs.shape: (32, 20000)\n",
      "inputs.dtype: <dtype: 'float32'>\n",
      "targets.shape: (32,)\n",
      "targets.dtype: <dtype: 'int32'>\n",
      "inputs[0]: tf.Tensor([1. 1. 1. ... 0. 0. 0.], shape=(20000,), dtype=float32)\n",
      "targets[0]: tf.Tensor(1, shape=(), dtype=int32)\n",
      "time: 272 ms (started: 2021-08-03 14:55:55 +08:00)\n"
     ]
    }
   ],
   "source": [
    "for inputs, targets in binary_1gram_train_ds:\n",
    "    print(\"inputs.shape:\", inputs.shape) #输入是 20,000 维向量的批次。\n",
    "    print(\"inputs.dtype:\", inputs.dtype)\n",
    "    print(\"targets.shape:\", targets.shape)\n",
    "    print(\"targets.dtype:\", targets.dtype)\n",
    "    print(\"inputs[0]:\", inputs[0]) #这些向量完全由 1 和 0 组成。\n",
    "    print(\"targets[0]:\", targets[0])\n",
    "    break"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "literary-riding",
   "metadata": {},
   "source": [
    "接下来，让我们编写一个可重用的模型构建函数，我们将在本节的所有实验中使用该函数。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "boolean-behavior",
   "metadata": {},
   "source": [
    "> 清单 11.7 我们的模型构建工具"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "id": "mexican-housing",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "time: 935 µs (started: 2021-08-03 15:20:32 +08:00)\n"
     ]
    }
   ],
   "source": [
    "from tensorflow import keras\n",
    "from tensorflow.keras import layers\n",
    "\n",
    "def get_model(max_tokens=20000, hidden_dim=16):\n",
    "    inputs = keras.Input(shape=(max_tokens,))\n",
    "    x = layers.Dense(hidden_dim, activation=\"relu\")(inputs)\n",
    "    x = layers.Dropout(0.5)(x)\n",
    "    outputs = layers.Dense(1, activation=\"sigmoid\")(x)\n",
    "    \n",
    "    model = keras.Model(inputs, outputs)\n",
    "    model.compile(optimizer=\"rmsprop\",\n",
    "                  loss=\"binary_crossentropy\",\n",
    "                  metrics=[\"accuracy\"])\n",
    "    return model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "id": "brown-witness",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"model\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "input_1 (InputLayer)         [(None, 20000)]           0         \n",
      "_________________________________________________________________\n",
      "dense (Dense)                (None, 16)                320016    \n",
      "_________________________________________________________________\n",
      "dropout (Dropout)            (None, 16)                0         \n",
      "_________________________________________________________________\n",
      "dense_1 (Dense)              (None, 1)                 17        \n",
      "=================================================================\n",
      "Total params: 320,033\n",
      "Trainable params: 320,033\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n",
      "time: 96.8 ms (started: 2021-08-03 15:20:53 +08:00)\n"
     ]
    }
   ],
   "source": [
    "model = get_model()\n",
    "model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "drawn-flexibility",
   "metadata": {},
   "source": [
    "最后，让我们训练和测试我们的模型："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "id": "departmental-manhattan",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/10\n",
      "625/625 [==============================] - 20s 24ms/step - loss: 0.3924 - accuracy: 0.8360 - val_loss: 0.2779 - val_accuracy: 0.8872\n",
      "Epoch 2/10\n",
      "625/625 [==============================] - 10s 16ms/step - loss: 0.2668 - accuracy: 0.9011 - val_loss: 0.2778 - val_accuracy: 0.8972\n",
      "Epoch 3/10\n",
      "625/625 [==============================] - 9s 14ms/step - loss: 0.2391 - accuracy: 0.9179 - val_loss: 0.2890 - val_accuracy: 0.8960\n",
      "Epoch 4/10\n",
      "625/625 [==============================] - 10s 16ms/step - loss: 0.2266 - accuracy: 0.9226 - val_loss: 0.3013 - val_accuracy: 0.8968\n",
      "Epoch 5/10\n",
      "625/625 [==============================] - 9s 15ms/step - loss: 0.2187 - accuracy: 0.9302 - val_loss: 0.3165 - val_accuracy: 0.8974\n",
      "Epoch 6/10\n",
      "625/625 [==============================] - 10s 16ms/step - loss: 0.2143 - accuracy: 0.9322 - val_loss: 0.3227 - val_accuracy: 0.8966\n",
      "Epoch 7/10\n",
      "625/625 [==============================] - 8s 14ms/step - loss: 0.2097 - accuracy: 0.9334 - val_loss: 0.3353 - val_accuracy: 0.8974\n",
      "Epoch 8/10\n",
      "625/625 [==============================] - 10s 16ms/step - loss: 0.2130 - accuracy: 0.9329 - val_loss: 0.3417 - val_accuracy: 0.8988\n",
      "Epoch 9/10\n",
      "625/625 [==============================] - 9s 14ms/step - loss: 0.2134 - accuracy: 0.9322 - val_loss: 0.3510 - val_accuracy: 0.8956\n",
      "Epoch 10/10\n",
      "625/625 [==============================] - 10s 16ms/step - loss: 0.2059 - accuracy: 0.9358 - val_loss: 0.3572 - val_accuracy: 0.8962\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<tensorflow.python.keras.callbacks.History at 0x7ff058025a60>"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "time: 1min 47s (started: 2021-08-03 15:23:21 +08:00)\n"
     ]
    }
   ],
   "source": [
    "# model = get_model()\n",
    "# model.summary()\n",
    "callbacks = [\n",
    "    keras.callbacks.ModelCheckpoint(\"binary_1gram.keras\",\n",
    "                                    save_best_only=True)]\n",
    "\n",
    "# 我们调用数据集将它们缓存在内存中：这样，在第一个 epoch 期间，我们只会 cache() 进行一次预处理，\n",
    "# 并且我们将在接下来的 epoch 中重用预处理过的文本。 只有当数据小到可以放入内存时才能这样做。\n",
    "model.fit(binary_1gram_train_ds.cache(),\n",
    "          validation_data=binary_1gram_val_ds.cache(),\n",
    "          epochs=10,\n",
    "          callbacks=callbacks)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "id": "impaired-hands",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "782/782 [==============================] - 13s 17ms/step - loss: 0.2891 - accuracy: 0.8892\n",
      "Test acc: 0.889\n",
      "time: 13.7 s (started: 2021-08-03 15:26:27 +08:00)\n"
     ]
    }
   ],
   "source": [
    "model = keras.models.load_model(\"binary_1gram.keras\")\n",
    "\n",
    "print(f\"Test acc: {model.evaluate(binary_1gram_test_ds)[1]:.3f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "binary-visitor",
   "metadata": {},
   "source": [
    "这使我们的测试准确率为 89.2%：不错！ 请注意，在这种情况下，由于数据集是一个平衡的二分类数据集（正样本与负样本一样多），我们可以在不训练实际模型的情况下达到的“朴素基线”只有 50%。 同时，在不利用外部数据的情况下，可以在该数据集上获得的最佳分数是大约 95% 的测试准确率。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "known-johnson",
   "metadata": {},
   "source": [
    "**二进制编码的 BIGRAMS**"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "unusual-minneapolis",
   "metadata": {},
   "source": [
    "当然，放弃词序是非常还原的，因为即使是原子概念也可以通过多个词来表达：“美国”一词传达的概念与将“国家”和“联合”这两个词分开的含义截然不同。 出于这个原因，您通常最终会通过查看 N-gram 而不是单个单词（最常见的是二元组）将本地顺序信息重新注入到您的词袋表示中。\n",
    "\n",
    "使用 bigrams，我们的句子变成：\n",
    "```\n",
    "{\"the\", \"the cat\", \"cat\", \"cat sat\", \"sat\",\n",
    " \"sat on\", \"on\", \"on the\", \"the mat\", \"mat\"}\n",
    " ```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "motivated-moscow",
   "metadata": {},
   "source": [
    "TextVectorization 层可以配置为返回任意 N-grams：bigrams、trigrams 等。只需传递一个参数，如下所示："
   ]
  },
  {
   "cell_type": "markdown",
   "id": "informal-reporter",
   "metadata": {},
   "source": [
    "> 清单 11.9 配置 TextVectorization 层以返回二元组"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "id": "hungry-manner",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "time: 19.2 ms (started: 2021-08-03 15:28:52 +08:00)\n"
     ]
    }
   ],
   "source": [
    "text_vectorization = TextVectorization(\n",
    "    ngrams=2,\n",
    "    max_tokens=20000,\n",
    "    output_mode=\"binary\",\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "small-singapore",
   "metadata": {},
   "source": [
    "让我们测试您的模型在此类二进制编码的二元组上训练时的表现："
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dying-valuation",
   "metadata": {},
   "source": [
    "> 清单 11.10 训练和测试二进制二元模型"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "id": "imperial-lexington",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"model_1\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "input_2 (InputLayer)         [(None, 20000)]           0         \n",
      "_________________________________________________________________\n",
      "dense_2 (Dense)              (None, 16)                320016    \n",
      "_________________________________________________________________\n",
      "dropout_1 (Dropout)          (None, 16)                0         \n",
      "_________________________________________________________________\n",
      "dense_3 (Dense)              (None, 1)                 17        \n",
      "=================================================================\n",
      "Total params: 320,033\n",
      "Trainable params: 320,033\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n",
      "Epoch 1/10\n",
      "625/625 [==============================] - 20s 31ms/step - loss: 0.3805 - accuracy: 0.8431 - val_loss: 0.2672 - val_accuracy: 0.9006\n",
      "Epoch 2/10\n",
      "625/625 [==============================] - 10s 16ms/step - loss: 0.2398 - accuracy: 0.9162 - val_loss: 0.2899 - val_accuracy: 0.8986\n",
      "Epoch 3/10\n",
      "625/625 [==============================] - 9s 14ms/step - loss: 0.2114 - accuracy: 0.9301 - val_loss: 0.2966 - val_accuracy: 0.9052\n",
      "Epoch 4/10\n",
      "625/625 [==============================] - 11s 17ms/step - loss: 0.1944 - accuracy: 0.9406 - val_loss: 0.3106 - val_accuracy: 0.9042\n",
      "Epoch 5/10\n",
      "625/625 [==============================] - 9s 15ms/step - loss: 0.1891 - accuracy: 0.9438 - val_loss: 0.3268 - val_accuracy: 0.9024\n",
      "Epoch 6/10\n",
      "625/625 [==============================] - 9s 14ms/step - loss: 0.1883 - accuracy: 0.9480 - val_loss: 0.3447 - val_accuracy: 0.8998\n",
      "Epoch 7/10\n",
      "625/625 [==============================] - 11s 17ms/step - loss: 0.1804 - accuracy: 0.9499 - val_loss: 0.3557 - val_accuracy: 0.8994\n",
      "Epoch 8/10\n",
      "625/625 [==============================] - 8s 13ms/step - loss: 0.1767 - accuracy: 0.9514 - val_loss: 0.3629 - val_accuracy: 0.8976\n",
      "Epoch 9/10\n",
      "625/625 [==============================] - 11s 17ms/step - loss: 0.1755 - accuracy: 0.9514 - val_loss: 0.3788 - val_accuracy: 0.8960\n",
      "Epoch 10/10\n",
      "625/625 [==============================] - 10s 16ms/step - loss: 0.1717 - accuracy: 0.9545 - val_loss: 0.3701 - val_accuracy: 0.8984\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<tensorflow.python.keras.callbacks.History at 0x7fef7db9d2b0>"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "time: 2min 22s (started: 2021-08-03 15:30:40 +08:00)\n"
     ]
    }
   ],
   "source": [
    "text_vectorization.adapt(text_only_train_ds)\n",
    "                         \n",
    "binary_2gram_train_ds = train_ds.map(lambda x, y: (text_vectorization(x), y))\n",
    "binary_2gram_val_ds = val_ds.map(lambda x, y: (text_vectorization(x), y))\n",
    "binary_2gram_test_ds = test_ds.map(lambda x, y: (text_vectorization(x), y))\n",
    "\n",
    "model = get_model()\n",
    "model.summary()\n",
    "\n",
    "callbacks = [keras.callbacks.ModelCheckpoint(\"binary_2gram.keras\",\n",
    "                                             save_best_only=True)]\n",
    "                         \n",
    "model.fit(binary_2gram_train_ds.cache(),\n",
    "          validation_data=binary_2gram_val_ds.cache(),\n",
    "          epochs=10,\n",
    "          callbacks=callbacks)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "id": "hungry-uniform",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "782/782 [==============================] - 19s 23ms/step - loss: 0.2653 - accuracy: 0.8973\n",
      "Test acc: 0.897\n",
      "time: 19.7 s (started: 2021-08-03 15:33:16 +08:00)\n"
     ]
    }
   ],
   "source": [
    "model = keras.models.load_model(\"binary_2gram.keras\")\n",
    "print(f\"Test acc: {model.evaluate(binary_2gram_test_ds)[1]:.3f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "macro-companion",
   "metadata": {},
   "source": [
    "我们现在获得了 90.4% 的测试准确率，这是一个显着的改进！ 事实证明局域顺序非常重要。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "interesting-atlas",
   "metadata": {},
   "source": [
    "**带有 TF-IDF 编码的 BIGRAMS**"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "challenging-spell",
   "metadata": {},
   "source": [
    "您还可以通过计算每个单词或 N-gram 出现的次数来为该表示添加更多信息，也就是说，通过获取文本上的单词的直方图：\n",
    "```\n",
    "{\"the\": 2, \"the cat\": 1, \"cat\": 1, \"cat sat\": 1, \"sat\": 1,\n",
    " \"sat on\": 1, \"on\": 1, \"on the\": 1, \"the mat: 1\", \"mat\": 1}\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "atomic-mineral",
   "metadata": {},
   "source": [
    "如果您在进行文本分类，那么了解某个单词在样本中出现的次数至关重要：无论情绪如何，任何足够长的电影评论都可能包含“terrible”一词，但评论中包含“terrible”一词的许多实例 很可能是负面的。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "rental-english",
   "metadata": {},
   "source": [
    "下面是如何使用 TextVectorization 层计算双字组出现的次数："
   ]
  },
  {
   "cell_type": "markdown",
   "id": "direct-wholesale",
   "metadata": {},
   "source": [
    "> 清单 11.11 配置 TextVectorization 层以返回标记计数"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "id": "funded-habitat",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "time: 5.51 ms (started: 2021-08-03 15:36:20 +08:00)\n"
     ]
    }
   ],
   "source": [
    "text_vectorization = TextVectorization(\n",
    "    ngrams=2,\n",
    "    max_tokens=20000,\n",
    "    output_mode=\"count\"\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bigger-expression",
   "metadata": {},
   "source": [
    "现在，当然，无论您在看什么电影评论，有些词肯定会比其他词更频繁地出现。文本是关于什么的。单词“the”、“a”、“is”和“are”将始终主导您的字数直方图，淹没其他单词——尽管在分类上下文中几乎是无用的特征。我们如何解决这个问题？\n",
    "\n",
    "您已经猜到了：通过规范化。我们可以通过减去均值并除以方差（在整个训练数据集中计算）来标准化字数。那是有道理的。除了……大多数向量化句子几乎完全由零组成（我们上面的示例具有 12 个非零条目和 19,988 个零条目），这是一种称为“稀疏性”的属性。这是一个很好的属性，因为它可以显着减少计算负载并降低过度拟合的风险。如果我们从每个特征中减去平均值，我们就会破坏稀疏性。因此，我们使用的任何归一化方案都应该是只除法。那么，我们应该使用什么作为分母呢？最佳实践是使用称为 TF-IDF 规范化的东西——TF-IDF 代表“词频，逆文档频率”。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "infrared-slope",
   "metadata": {},
   "source": [
    "> **理解 TF-IDF 归一化**\n",
    "> \n",
    "> 给定术语在文档中出现的次数越多，该术语对于理解文档的内容就越重要。 同时，术语在数据集中所有文档中出现的频率也很重要：出现在几乎每个文档中的术语（如“the”或“a”）并不是特别有用，而仅出现在 所有文本的一小部分（如“Herzog”）非常独特，因此很重要。 TF-IDF 是一个融合了这两个想法的指标。 它通过获取“术语频率”、该术语在当前文档中出现的次数并将其除以“文档频率”的度量来对给定术语进行加权，从而估计该术语在数据集中出现的频率。 您可以将其计算为：\n",
    ">\n",
    "> ```\n",
    "def tfidf(term, document, dataset):\n",
    "    term_freq = document.count(term)\n",
    "    doc_freq = math.log(sum(doc.count(term) for doc in dataset) + 1)\n",
    "    return term_freq / doc_freq\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "global-ghana",
   "metadata": {},
   "source": [
    "TF-IDF 非常常见，以至于它内置于 TextVectorization 层中。 开始使用它所需要做的就是将 output_mode 参数切换为："
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fossil-broadcasting",
   "metadata": {},
   "source": [
    "> 清单 11.12 配置 TextVectorization 层以返回 TF-IDF 加权输出"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "id": "resistant-robin",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "time: 26.1 ms (started: 2021-08-03 15:42:00 +08:00)\n"
     ]
    }
   ],
   "source": [
    "text_vectorization = TextVectorization(\n",
    "    ngrams=2,\n",
    "    max_tokens=20000,\n",
    "    output_mode=\"tf-idf\",\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "looking-retention",
   "metadata": {},
   "source": [
    "> 清单 11.13 训练和测试 TF-IDF 双元组模型"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "id": "warming-muslim",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"model_2\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "input_3 (InputLayer)         [(None, 20000)]           0         \n",
      "_________________________________________________________________\n",
      "dense_4 (Dense)              (None, 16)                320016    \n",
      "_________________________________________________________________\n",
      "dropout_2 (Dropout)          (None, 16)                0         \n",
      "_________________________________________________________________\n",
      "dense_5 (Dense)              (None, 1)                 17        \n",
      "=================================================================\n",
      "Total params: 320,033\n",
      "Trainable params: 320,033\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n",
      "Epoch 1/10\n",
      "625/625 [==============================] - 23s 36ms/step - loss: 0.4763 - accuracy: 0.7936 - val_loss: 0.3335 - val_accuracy: 0.8828\n",
      "Epoch 2/10\n",
      "625/625 [==============================] - 10s 15ms/step - loss: 0.2987 - accuracy: 0.8828 - val_loss: 0.2927 - val_accuracy: 0.8890\n",
      "Epoch 3/10\n",
      "625/625 [==============================] - 10s 16ms/step - loss: 0.2859 - accuracy: 0.8852 - val_loss: 0.3209 - val_accuracy: 0.8652\n",
      "Epoch 4/10\n",
      "625/625 [==============================] - 9s 14ms/step - loss: 0.2651 - accuracy: 0.8939 - val_loss: 0.3185 - val_accuracy: 0.8720\n",
      "Epoch 5/10\n",
      "625/625 [==============================] - 9s 14ms/step - loss: 0.2502 - accuracy: 0.8964 - val_loss: 0.3223 - val_accuracy: 0.8800\n",
      "Epoch 6/10\n",
      "625/625 [==============================] - 10s 16ms/step - loss: 0.2348 - accuracy: 0.9043 - val_loss: 0.3377 - val_accuracy: 0.8726\n",
      "Epoch 7/10\n",
      "625/625 [==============================] - 12s 20ms/step - loss: 0.2286 - accuracy: 0.9080 - val_loss: 0.3514 - val_accuracy: 0.8666\n",
      "Epoch 8/10\n",
      "625/625 [==============================] - 9s 15ms/step - loss: 0.2290 - accuracy: 0.9054 - val_loss: 0.3394 - val_accuracy: 0.8756\n",
      "Epoch 9/10\n",
      "625/625 [==============================] - 9s 14ms/step - loss: 0.2197 - accuracy: 0.9106 - val_loss: 0.3460 - val_accuracy: 0.8714\n",
      "Epoch 10/10\n",
      "625/625 [==============================] - 11s 18ms/step - loss: 0.2119 - accuracy: 0.9100 - val_loss: 0.3664 - val_accuracy: 0.8758\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<tensorflow.python.keras.callbacks.History at 0x7fef7d6e1d90>"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "time: 2min 30s (started: 2021-08-03 15:42:56 +08:00)\n"
     ]
    }
   ],
   "source": [
    "text_vectorization.adapt(text_only_train_ds) \n",
    "\n",
    "tfidf_2gram_train_ds = train_ds.map(lambda x, y: (text_vectorization(x), y))\n",
    "tfidf_2gram_val_ds = val_ds.map(lambda x, y: (text_vectorization(x), y))\n",
    "tfidf_2gram_test_ds = test_ds.map(lambda x, y: (text_vectorization(x), y))\n",
    "\n",
    "model = get_model()\n",
    "model.summary()\n",
    "\n",
    "callbacks = [keras.callbacks.ModelCheckpoint(\"tfidf_2gram.keras\",\n",
    "                                             save_best_only=True)]\n",
    "\n",
    "model.fit(tfidf_2gram_train_ds.cache(),\n",
    "          validation_data=tfidf_2gram_val_ds.cache(),\n",
    "          epochs=10,\n",
    "          callbacks=callbacks)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "filled-franchise",
   "metadata": {},
   "outputs": [],
   "source": [
    "model = keras.models.load_model(\"tfidf_2gram.keras\")\n",
    "print(f\"Test acc: {model.evaluate(tfidf_2gram_test_ds)[1]:.3f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "hearing-audit",
   "metadata": {},
   "source": [
    "这让我们在 IMDB 分类任务上获得了 89.8% 的测试准确率：在这种情况下它似乎不是特别有用。 但是，对于许多文本分类数据集，与纯二进制编码相比，使用 TF-IDF 时通常会看到一个百分点的增加。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "careful-respect",
   "metadata": {},
   "source": [
    "> **导出处理原始字符串的模型**\n",
    ">\n",
    "> 在上面的示例中，我们将文本标准化、拆分和索引作为管道的一部分。 但是如果我们想导出一个独立于这个管道的独立模型 tf.data，我们应该确保它包含自己的文本预处理（否则，你必须在生产环境中重新实现，这可能具有挑战性或可能导致 训练数据和生产数据之间的细微差异）。 谢天谢地，这很容易。\n",
    "> \n",
    "> 只需创建一个新模型，重用您的 TextVectorization 层，并将您刚刚训练的模型添加到其中：\n",
    ">```python\n",
    "inputs = keras.Input(shape=(1,), dtype=\"string\") \n",
    "processed_inputs = text_vectorization(inputs)\n",
    "outputs = model(processed_inputs)\n",
    "inference_model = keras.Model(inputs, outputs)\n",
    ">```\n",
    ">"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "centered-container",
   "metadata": {},
   "source": [
    "> 生成的模型可以处理成批的原始字符串：\n",
    ">\n",
    "> ```\n",
    "import tensorflow as tf\n",
    "raw_text_data = tf.convert_to_tensor([[\"That was an excellent movie, I loved it.\"],])\n",
    "predictions = inference_model(raw_text_data)\n",
    "print(f\"{float(predictions[0] * 100):.2f} percent positive\")\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "hollywood-genre",
   "metadata": {},
   "source": [
    "### 将单词作为序列处理：序列模型方法"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "transparent-address",
   "metadata": {},
   "source": [
    "过去的几个例子清楚地表明词序很重要：基于顺序的特征的手动工程，例如二元组，产生了很好的准确度提升。 现在，请记住：深度学习的历史是从手动特征工程转向让模型仅通过接触数据来学习自己的特征。 如果我们不是手动制作基于顺序的特征，而是将模型暴露给原始单词序列，然后让它自己找出这些特征，会怎么样？ 这就是序列模型的意义所在。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "considerable-anxiety",
   "metadata": {},
   "source": [
    "要实现序列模型，您首先要将输入样本表示为整数索引序列（一个整数代表一个单词）。 然后，您将每个整数映射到一个向量，以获得向量序列。 最后，您将这些向量序列输入到一个层堆栈中，这些层可以将来自相邻向量的特征相互关联，例如 1D 卷积网络、RNN 或 Transformer。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "caring-medium",
   "metadata": {},
   "source": [
    "在 2016-2017 年左右的一段时间里，双向 RNN（特别是双向 LSTM）被认为是序列建模的最新技术。 由于您已经熟悉这种架构，这就是我们将在下面的第一个序列模型示例中使用的架构。 然而，如今，序列建模几乎普遍使用 Transformer 完成，我们将在稍后介绍。 奇怪的是，一维卷积网络在 NLP 中从来都不是很流行——尽管根据我自己的经验，深度可分离的一维卷积的残余堆栈通常可以实现与双向 LSTM 相当的性能，同时大大降低了计算成本。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "wrapped-competition",
   "metadata": {},
   "source": [
    "**第一个实际例子**"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "iraqi-strike",
   "metadata": {},
   "source": [
    "让我们在实践中尝试第一个序列模型。 首先，让我们准备返回整数序列的数据集。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "conservative-retrieval",
   "metadata": {},
   "source": [
    "> 清单 11.14 准备整数序列数据集"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "opponent-aviation",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "time: 7.61 s (started: 2021-08-04 14:18:16 +08:00)\n"
     ]
    }
   ],
   "source": [
    "from tensorflow.keras.layers.experimental.preprocessing import TextVectorization\n",
    "\n",
    "# 为了保持可管理的输入大小，我们将在前 600 个单词之后截断输入。 \n",
    "# 这是一个合理的选择，因为平均评论长度为 233 字，只有 5% 的评论超过 600 字形成连续批次。\n",
    "max_length = 600\n",
    "max_tokens = 20000\n",
    "\n",
    "text_vectorization = TextVectorization(max_tokens=max_tokens,\n",
    "                                       output_mode=\"int\",\n",
    "                                       output_sequence_length=max_length, )\n",
    "\n",
    "# 准备一个仅产生原始文本输入（无标签）的数据集。\n",
    "text_only_train_ds = train_ds.map(lambda x, y: x) \n",
    "# 通过adapter() 方法，使用该数据集来索引数据集词汇表。\n",
    "text_vectorization.adapt(text_only_train_ds) \n",
    "\n",
    "int_train_ds = train_ds.map(lambda x, y: (text_vectorization(x), y))\n",
    "int_val_ds = val_ds.map(lambda x, y: (text_vectorization(x), y))\n",
    "int_test_ds = test_ds.map(lambda x, y: (text_vectorization(x), y))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "successful-internet",
   "metadata": {},
   "source": [
    "接下来，让我们制作一个模型。 我们可以使用的将整数序列转换为向量序列的最简单工具是对整数进行单热编码（每个维度代表词汇表中一个可能的术语）。 在这些 one-hot 向量之上，我们将添加一个简单的双向 LSTM。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "internal-headset",
   "metadata": {},
   "source": [
    "> 清单 11.15 建立在 one-hot 编码向量序列之上的序列模型"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "intended-solomon",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"model\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "input_1 (InputLayer)         [(None, None)]            0         \n",
      "_________________________________________________________________\n",
      "tf.one_hot (TFOpLambda)      (None, None, 20000)       0         \n",
      "_________________________________________________________________\n",
      "bidirectional (Bidirectional (None, 64)                5128448   \n",
      "_________________________________________________________________\n",
      "dropout (Dropout)            (None, 64)                0         \n",
      "_________________________________________________________________\n",
      "dense (Dense)                (None, 1)                 65        \n",
      "=================================================================\n",
      "Total params: 5,128,513\n",
      "Trainable params: 5,128,513\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n",
      "time: 1.36 s (started: 2021-08-04 14:18:27 +08:00)\n"
     ]
    }
   ],
   "source": [
    "import tensorflow as tf\n",
    "from tensorflow.keras import layers\n",
    " \n",
    "# 一个输入是 100 个整数的序列。\n",
    "inputs = keras.Input(shape=(None,), dtype=\"int64\") \n",
    "\n",
    "# 将整数编码为 20,000 维的二进制向量。\n",
    "embedded = tf.one_hot(inputs, depth=max_tokens)\n",
    "\n",
    "# 添加双向 LSTM。\n",
    "x = layers.Bidirectional(layers.LSTM(32))(embedded) \n",
    "x = layers.Dropout(0.5)(x)\n",
    "# 最后，添加一个分类层。\n",
    "outputs = layers.Dense(1, activation=\"sigmoid\")(x) \n",
    "\n",
    "model = keras.Model(inputs, outputs)\n",
    "\n",
    "model.compile(optimizer=\"rmsprop\",\n",
    "              loss=\"binary_crossentropy\",\n",
    "              metrics=[\"accuracy\"])\n",
    "\n",
    "model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "introductory-forward",
   "metadata": {},
   "source": [
    "现在，让我们训练我们的模型。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "political-demand",
   "metadata": {},
   "source": [
    "> 清单 11.16 训练第一个基本序列模型"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "exotic-glucose",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/10\n",
      "625/625 [==============================] - 122s 177ms/step - loss: 0.5218 - accuracy: 0.7531 - val_loss: 0.3555 - val_accuracy: 0.8580\n",
      "Epoch 2/10\n",
      "625/625 [==============================] - 108s 173ms/step - loss: 0.3560 - accuracy: 0.8723 - val_loss: 0.3107 - val_accuracy: 0.8744\n",
      "Epoch 3/10\n",
      "625/625 [==============================] - 109s 174ms/step - loss: 0.2767 - accuracy: 0.9032 - val_loss: 0.2732 - val_accuracy: 0.8908\n",
      "Epoch 4/10\n",
      "625/625 [==============================] - 109s 175ms/step - loss: 0.2305 - accuracy: 0.9218 - val_loss: 0.3045 - val_accuracy: 0.8784\n",
      "Epoch 5/10\n",
      "625/625 [==============================] - 108s 172ms/step - loss: 0.2115 - accuracy: 0.9280 - val_loss: 0.3783 - val_accuracy: 0.8292\n",
      "Epoch 6/10\n",
      "625/625 [==============================] - 110s 176ms/step - loss: 0.1820 - accuracy: 0.9385 - val_loss: 0.3038 - val_accuracy: 0.8900\n",
      "Epoch 7/10\n",
      "625/625 [==============================] - 109s 175ms/step - loss: 0.1637 - accuracy: 0.9459 - val_loss: 0.3050 - val_accuracy: 0.8790\n",
      "Epoch 8/10\n",
      "625/625 [==============================] - 108s 174ms/step - loss: 0.1441 - accuracy: 0.9534 - val_loss: 0.3780 - val_accuracy: 0.8862\n",
      "Epoch 9/10\n",
      "625/625 [==============================] - 109s 174ms/step - loss: 0.1287 - accuracy: 0.9567 - val_loss: 0.3749 - val_accuracy: 0.8874\n",
      "Epoch 10/10\n",
      "625/625 [==============================] - 108s 173ms/step - loss: 0.1157 - accuracy: 0.9620 - val_loss: 0.3501 - val_accuracy: 0.8608\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<tensorflow.python.keras.callbacks.History at 0x7f64c6cb46a0>"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "time: 18min 25s (started: 2021-08-04 14:32:40 +08:00)\n"
     ]
    }
   ],
   "source": [
    "callbacks = [keras.callbacks.ModelCheckpoint(\"one_hot_bidir_lstm.keras\",\n",
    "                                             save_best_only=True)]\n",
    "\n",
    "model.fit(int_train_ds, validation_data=int_val_ds, epochs=10, callbacks=callbacks)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "heated-carter",
   "metadata": {},
   "outputs": [],
   "source": [
    "model = keras.models.load_model(\"one_hot_bidir_lstm.keras\")\n",
    "print(f\"Test acc: {model.evaluate(int_test_ds)[1]:.3f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "million-biodiversity",
   "metadata": {},
   "source": [
    "第一个观察：这个模型训练非常慢，尤其是与上一节的轻量级模型相比。 这是因为我们的输入非常大：每个输入样本都被编码为一个大小为 (600, 20000) 的矩阵（每个样本 600 个单词，20000 个可能的单词）。 对于单个电影评论来说，这是 12,000,000 个浮点数。 我们的双向 LSTM 有很多工作要做。 其次，该模型只能达到 87% 的测试准确率——它的表现几乎不如我们的（非常快的）二元一元模型。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "stretch-dairy",
   "metadata": {},
   "source": [
    "显然，使用 one-hot 编码将单词转换为向量，这是我们能做的最简单的事情，并不是一个好主意。 有一个更好的方法：词嵌入。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dirty-external",
   "metadata": {},
   "source": [
    "**理解词嵌入**"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "lesser-acrylic",
   "metadata": {},
   "source": [
    "至关重要的是，当您通过 one-hot 编码对某些内容进行编码时，您正在做出特征工程决策。您正在向模型中注入关于特征空间结构的基本假设。该假设是您编码的不同标记都彼此独立：实际上，one-hot 向量都彼此正交。就文字而言，这种假设显然是错误的。单词形成一个结构化的空间：它们彼此共享信息。在大多数句子中，“电影”和“电影”这两个词可以互换，因此表示“电影”的向量不应与表示“电影”的向量正交——它们应该是相同的向量，或者足够接近。\n",
    "\n",
    "为了更抽象一点，两个词向量之间的几何关系应该反映这些词之间的语义关系。例如，在一个合理的词向量空间中，你会期望同义词被嵌入到相似的词向量中，一般来说，你会期望任意两个词向量之间的几何距离（例如余弦距离或 L2 距离）与相关词之间的“语义距离”。表示不同事物的词应该彼此远离，而相关词应该靠得更近。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "defined-basket",
   "metadata": {},
   "source": [
    "词嵌入是词的向量表示，可以实现这一点：它们将人类语言映射到结构化的几何空间中。\n",
    "\n",
    "虽然通过 one-hot 编码获得的向量是二进制的、稀疏的（主要由零组成）和非常高维的（与词汇表中的单词数量相同），但词嵌入是低维浮点向量（即 是密集向量，而不是稀疏向量）； 见图 11.2。 在处理非常大的词汇表时，通常会看到 256 维、512 维或 1,024 维的词嵌入。 另一方面，one-hot 编码词通常会导致 20,000 维或更大的向量（在这种情况下捕获 20,000 个标记的词汇表）。 因此，词嵌入将更多信息打包到更少的维度中。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "liable-provincial",
   "metadata": {},
   "source": [
    "![](https://tva1.sinaimg.cn/large/008i3skNgy1gt4r73737dj30my0rgmys.jpg)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "secondary-atlantic",
   "metadata": {},
   "source": [
    "除了表示之外，词嵌入也是表示，它们的结构是从数据中学习到的密集结构。 相似的词被嵌入在附近的位置，而且嵌入空间中的特定词是有意义的。 为了更清楚地说明这一点，让我们看一个具体的方向示例。\n",
    "\n",
    "在图 11.3 中，四个词嵌入在 2D 平面上：cat、dog、wofl 和 Tiger。 使用我们在这里选择的向量表示，这些词之间的一些语义关系可以编码为几何变换。 例如，同一个向量允许我们从猫到老虎，从狗到狼：这个向量可以解释为“从宠物到野生动物”的向量。 类似地，另一个向量让我们从狗到猫，从狼到老虎，这可以解释为“从犬到猫”的向量。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "swiss-briefing",
   "metadata": {},
   "source": [
    "![](https://tva1.sinaimg.cn/large/008i3skNgy1gt4r9zfarpj30xi0lygmb.jpg)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "collected-involvement",
   "metadata": {},
   "source": [
    "在现实世界的词嵌入空间中，有意义的几何变换的常见示例是“性别”向量和“复数”向量。 例如，通过向向量“king”添加“female”向量，我们得到向量“queen”。 通过添加“复数”向量，我们得到“国王”。 词嵌入空间通常具有数千个此类可解释且可能有用的向量。\n",
    "\n",
    "让我们看看如何在实践中使用这样的嵌入空间。 有两种方法可以获得词嵌入：\n",
    "* 与您关心的主要任务（例如文档分类或情感预测）一起学习词嵌入。 在此设置中，您从随机词向量开始，然后以与学习神经网络权重相同的方式学习词向量。\n",
    "* 将使用与您尝试解决的任务不同的机器学习任务预先计算的词嵌入加载到您的模型中。 这些被称为预训练词嵌入。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "current-diana",
   "metadata": {},
   "source": [
    "让我们回顾一下这些方法中的每一种。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "linear-album",
   "metadata": {},
   "source": [
    "**使用Embedding层学习词嵌入**"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "rapid-recruitment",
   "metadata": {},
   "source": [
    "是否有一些理想的词嵌入空间可以完美地映射人类语言并可以用于任何自然语言处理任务？ 可能，但我们还没有计算出任何类似的东西。 此外，没有人类语言这样的东西——有许多不同的语言，它们彼此并不同构，因为一种语言是特定文化和特定语境的反映。 但更实用的是，什么是一个好的词嵌入空间在很大程度上取决于你的任务：英语电影评论情感分析模型的完美词嵌入空间可能与英语法律的完美嵌入空间不同。 文档分类模型，因为某些语义关系的重要性因任务而异。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "czech-worth",
   "metadata": {},
   "source": [
    "因此，为每个新任务学习一个新的嵌入空间是合理的。 幸运的是，反向传播使这变得容易，而 Keras 使这变得更加容易。 这是关于学习层的权重：Embedding层。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "authentic-difficulty",
   "metadata": {},
   "source": [
    "> 清单 11.17 实例化一个层"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "upset-conservative",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "time: 7.94 ms (started: 2021-08-04 15:02:49 +08:00)\n"
     ]
    }
   ],
   "source": [
    "# 嵌入层至少有两个参数：可能的标记数量和嵌入的维度（这里是 256）。\n",
    "embedding_layer = layers.Embedding(input_dim=max_tokens, output_dim=256)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "thirty-amino",
   "metadata": {},
   "source": [
    "该层最好理解为将整数索引（代表特定单词的嵌入）映射到密集向量的字典。 它接受整数作为输入，在内部字典中查找这些整数，并返回相关的向量。 它实际上是一个字典查找（见图 11.4）"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "responsible-midnight",
   "metadata": {},
   "source": [
    "![](https://tva1.sinaimg.cn/large/008i3skNgy1gt4rui2pgkj31fw052t9c.jpg)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "mounted-mixer",
   "metadata": {},
   "source": [
    "嵌入层将形状为 (batch_size, sequence_length) 的二维整数张量作为输入，其中每个条目都是一个整数序列。 然后该层返回一个形状为（batch_size、sequence_length、embedding_dimensionality）的 3D 浮点张量。\n",
    "\n",
    "当你实例化一个层时，它的权重（它的内部标记向量字典）嵌入最初是随机的，就像任何其他层一样。 在训练期间，这些词向量通过反向传播逐渐调整，将空间构造成下游模型可以利用的东西。 一旦经过充分训练，嵌入空间将显示出很多结构——一种专门用于训练模型的特定问题的结构。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "intelligent-sailing",
   "metadata": {},
   "source": [
    "让我们构建一个包含嵌入层的模型，并在我们的任务中对其进行基准测试："
   ]
  },
  {
   "cell_type": "markdown",
   "id": "separate-helicopter",
   "metadata": {},
   "source": [
    "> 清单 11.18 使用从头开始训练的嵌入层的模型"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "furnished-transcription",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"model_1\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "input_2 (InputLayer)         [(None, None)]            0         \n",
      "_________________________________________________________________\n",
      "embedding_1 (Embedding)      (None, None, 256)         5120000   \n",
      "_________________________________________________________________\n",
      "bidirectional_1 (Bidirection (None, 64)                73984     \n",
      "_________________________________________________________________\n",
      "dropout_1 (Dropout)          (None, 64)                0         \n",
      "_________________________________________________________________\n",
      "dense_1 (Dense)              (None, 1)                 65        \n",
      "=================================================================\n",
      "Total params: 5,194,049\n",
      "Trainable params: 5,194,049\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n",
      "Epoch 1/10\n",
      "625/625 [==============================] - 105s 153ms/step - loss: 0.4758 - accuracy: 0.7828 - val_loss: 0.3260 - val_accuracy: 0.8728\n",
      "Epoch 2/10\n",
      "625/625 [==============================] - 95s 152ms/step - loss: 0.3080 - accuracy: 0.8875 - val_loss: 0.3038 - val_accuracy: 0.8858\n",
      "Epoch 3/10\n",
      "625/625 [==============================] - 94s 150ms/step - loss: 0.2405 - accuracy: 0.9151 - val_loss: 0.3081 - val_accuracy: 0.8836\n",
      "Epoch 4/10\n",
      "625/625 [==============================] - 94s 150ms/step - loss: 0.1916 - accuracy: 0.9339 - val_loss: 0.3977 - val_accuracy: 0.8662\n",
      "Epoch 5/10\n",
      "625/625 [==============================] - 94s 151ms/step - loss: 0.1617 - accuracy: 0.9453 - val_loss: 0.3507 - val_accuracy: 0.8846\n",
      "Epoch 6/10\n",
      "625/625 [==============================] - 93s 148ms/step - loss: 0.1435 - accuracy: 0.9527 - val_loss: 0.6819 - val_accuracy: 0.8116\n",
      "Epoch 7/10\n",
      "625/625 [==============================] - 94s 150ms/step - loss: 0.1229 - accuracy: 0.9596 - val_loss: 0.3703 - val_accuracy: 0.8792\n",
      "Epoch 8/10\n",
      "625/625 [==============================] - 93s 149ms/step - loss: 0.0976 - accuracy: 0.9697 - val_loss: 0.4051 - val_accuracy: 0.8692\n",
      "Epoch 9/10\n",
      "625/625 [==============================] - 93s 149ms/step - loss: 0.0797 - accuracy: 0.9748 - val_loss: 0.4344 - val_accuracy: 0.8804\n",
      "Epoch 10/10\n",
      "625/625 [==============================] - 94s 150ms/step - loss: 0.0665 - accuracy: 0.9785 - val_loss: 0.5153 - val_accuracy: 0.8792\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<tensorflow.python.keras.callbacks.History at 0x7f62e8089970>"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "time: 15min 54s (started: 2021-08-04 15:52:10 +08:00)\n"
     ]
    }
   ],
   "source": [
    "inputs = keras.Input(shape=(None,), dtype=\"int64\")\n",
    "\n",
    "embedded = layers.Embedding(input_dim=max_tokens, output_dim=256)(inputs)\n",
    "\n",
    "x = layers.Bidirectional(layers.LSTM(32))(embedded)\n",
    "x = layers.Dropout(0.5)(x)\n",
    "outputs = layers.Dense(1, activation=\"sigmoid\")(x)\n",
    "\n",
    "model = keras.Model(inputs, outputs)\n",
    "\n",
    "model.compile(optimizer=\"rmsprop\",\n",
    "              loss=\"binary_crossentropy\",\n",
    "              metrics=[\"accuracy\"])\n",
    "\n",
    "model.summary()\n",
    "\n",
    "callbacks = [keras.callbacks.ModelCheckpoint(\"embeddings_bidir_gru.keras\",\n",
    "                                             save_best_only=True)\n",
    "            ]\n",
    "\n",
    "model.fit(int_train_ds, validation_data=int_val_ds, epochs=10, callbacks=callbacks)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "quality-ridge",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "782/782 [==============================] - 35s 40ms/step - loss: 0.3335 - accuracy: 0.8675\n",
      "Test acc: 0.868\n",
      "time: 37.4 s (started: 2021-08-04 16:08:04 +08:00)\n"
     ]
    }
   ],
   "source": [
    "model = keras.models.load_model(\"embeddings_bidir_gru.keras\")\n",
    "print(f\"Test acc: {model.evaluate(int_test_ds)[1]:.3f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "scheduled-choir",
   "metadata": {},
   "source": [
    "它的训练速度比 one-hot 模型快得多（因为 LSTM 只需处理 256 维向量而不是 20,000 维），并且其测试准确率相当（87%）。 然而，我们离我们的基本二元模型的结果还有一段距离。 部分原因很简单，模型查看的数据略少：bigram 模型处理完整评论，而我们的序列模型在 600 个单词后截断序列。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "defensive-apartment",
   "metadata": {},
   "source": [
    "**了解padding和masking**\n",
    "\n",
    "这里稍微影响模型性能的一件事是我们的输入序列全是零。 这来自于我们在 TextVectorization max_length（等于 600）中使用 output_sequence_length=max_length 选项：超过 600 个标记的句子被截断为 600 个标记的长度，而少于 600 个标记的句子在末尾用零填充，以便 它们可以与其他序列连接在一起以形成连续的批次。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bizarre-playlist",
   "metadata": {},
   "source": [
    "我们使用双向 RNN：并行运行的两个 RNN 层，一个以自然顺序处理令牌，另一个以相反的方式处理相同的令牌。 以自然顺序查看标记的 RNN 将在最后一次迭代中只看到编码填充的向量——如果原始句子很短，可能会进行数百次迭代。 当 RNN 暴露于这些无意义的输入时，存储在 RNN 内部状态中的信息将逐渐消失。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "valued-sixth",
   "metadata": {},
   "source": [
    "我们需要一些方法来告诉 RNN 它应该跳过这些迭代。 有一个 API：*masking*。\n",
    "\n",
    "该层能够生成与其输入数据相对应的“掩码”。 这个嵌入掩码是一个由 1 和 0（或 True/False 布尔值）组成的张量，形状为 (batch_size, sequence_length)，其中 entry mask[i, t] 指示应该跳过或不跳过样本 i 的时间步长 t（时间步长将 如果 mask[i, t] 为 0 或 False，则跳过，否则进行处理）。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "affecting-vector",
   "metadata": {},
   "source": [
    "默认情况下，此选项未激活 - 通过将 mask_zero=True 传递给您的嵌入层来打开它。 您可以通过 compute_mask() 方法检索掩码："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "underlying-element",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "time: 8.05 ms (started: 2021-08-04 16:15:13 +08:00)\n"
     ]
    }
   ],
   "source": [
    "embedding_layer = layers.Embedding(input_dim=10, output_dim=256, mask_zero=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "damaged-toronto",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "time: 735 µs (started: 2021-08-04 16:16:42 +08:00)\n"
     ]
    }
   ],
   "source": [
    "some_input = [[4, 3, 2, 1, 0, 0, 0],\n",
    "              [5, 4, 3, 2, 1, 0, 0],\n",
    "              [2, 1, 0, 0, 0, 0, 0]]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "affiliated-process",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(3, 7), dtype=bool, numpy=\n",
       "array([[ True,  True,  True,  True, False, False, False],\n",
       "       [ True,  True,  True,  True,  True, False, False],\n",
       "       [ True,  True, False, False, False, False, False]])>"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "time: 6.08 ms (started: 2021-08-04 16:17:50 +08:00)\n"
     ]
    }
   ],
   "source": [
    "mask = embedding_layer.compute_mask(some_input)\n",
    "mask"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "polyphonic-samoa",
   "metadata": {},
   "source": [
    "在实践中，您几乎不需要手动管理mask。 相反，Keras 会自动将掩码与其代表的序列一起传递给能够处理它的每个层。 RNN 层将使用此掩码来跳过掩码步骤。 如果您的模型返回整个序列，那么损失函数也将使用掩码来跳过输出序列中的掩码步骤。\n",
    "\n",
    "让我们尝试在启用掩码的情况下重新训练我们的模型："
   ]
  },
  {
   "cell_type": "markdown",
   "id": "similar-simon",
   "metadata": {},
   "source": [
    "> 清单 11.19 使用从头开始训练的嵌入层的模型，启用了掩码："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "flying-nightmare",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"model_2\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "input_3 (InputLayer)         [(None, None)]            0         \n",
      "_________________________________________________________________\n",
      "embedding_3 (Embedding)      (None, None, 256)         5120000   \n",
      "_________________________________________________________________\n",
      "bidirectional_2 (Bidirection (None, 64)                73984     \n",
      "_________________________________________________________________\n",
      "dropout_2 (Dropout)          (None, 64)                0         \n",
      "_________________________________________________________________\n",
      "dense_2 (Dense)              (None, 1)                 65        \n",
      "=================================================================\n",
      "Total params: 5,194,049\n",
      "Trainable params: 5,194,049\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n",
      "Epoch 1/10\n",
      "625/625 [==============================] - 128s 181ms/step - loss: 0.3991 - accuracy: 0.8181 - val_loss: 0.2815 - val_accuracy: 0.8864\n",
      "Epoch 2/10\n",
      "625/625 [==============================] - 106s 168ms/step - loss: 0.2288 - accuracy: 0.9115 - val_loss: 0.2615 - val_accuracy: 0.8942\n",
      "Epoch 3/10\n",
      "625/625 [==============================] - 111s 177ms/step - loss: 0.1660 - accuracy: 0.9384 - val_loss: 0.2834 - val_accuracy: 0.8820\n",
      "Epoch 4/10\n",
      "625/625 [==============================] - 107s 171ms/step - loss: 0.1232 - accuracy: 0.9557 - val_loss: 0.3084 - val_accuracy: 0.8882\n",
      "Epoch 5/10\n",
      "625/625 [==============================] - 110s 175ms/step - loss: 0.0898 - accuracy: 0.9695 - val_loss: 0.3796 - val_accuracy: 0.8886\n",
      "Epoch 6/10\n",
      "625/625 [==============================] - 108s 173ms/step - loss: 0.0651 - accuracy: 0.9782 - val_loss: 0.4353 - val_accuracy: 0.8724\n",
      "Epoch 7/10\n",
      "625/625 [==============================] - 106s 169ms/step - loss: 0.0486 - accuracy: 0.9844 - val_loss: 0.4680 - val_accuracy: 0.8654\n",
      "Epoch 8/10\n",
      "625/625 [==============================] - 116s 186ms/step - loss: 0.0346 - accuracy: 0.9893 - val_loss: 0.5436 - val_accuracy: 0.8598\n",
      "Epoch 9/10\n",
      "625/625 [==============================] - 104s 166ms/step - loss: 0.0259 - accuracy: 0.9919 - val_loss: 0.5030 - val_accuracy: 0.8686\n",
      "Epoch 10/10\n",
      "625/625 [==============================] - 108s 173ms/step - loss: 0.0179 - accuracy: 0.9948 - val_loss: 0.6068 - val_accuracy: 0.8742\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<tensorflow.python.keras.callbacks.History at 0x7f60c041e340>"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "time: 19min 9s (started: 2021-08-04 16:29:36 +08:00)\n"
     ]
    }
   ],
   "source": [
    "inputs = keras.Input(shape=(None,), dtype=\"int64\")\n",
    "\n",
    "embedded = layers.Embedding(input_dim=max_tokens, \n",
    "                            output_dim=256, \n",
    "                            mask_zero=True)(inputs)\n",
    "\n",
    "x = layers.Bidirectional(layers.LSTM(32))(embedded)\n",
    "x = layers.Dropout(0.5)(x)\n",
    "outputs = layers.Dense(1, activation=\"sigmoid\")(x)\n",
    "\n",
    "model = keras.Model(inputs, outputs)\n",
    "\n",
    "model.compile(optimizer=\"rmsprop\",\n",
    "              loss=\"binary_crossentropy\",\n",
    "              metrics=[\"accuracy\"])\n",
    "\n",
    "model.summary()\n",
    "\n",
    "callbacks = [\n",
    "    keras.callbacks.ModelCheckpoint(\"embeddings_bidir_gru_with_masking.keras\",\n",
    "                                    save_best_only=True)]\n",
    "\n",
    "model.fit(int_train_ds, validation_data=int_val_ds, epochs=10, callbacks=callbacks)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "sunset-hampton",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "782/782 [==============================] - 43s 49ms/step - loss: 0.2894 - accuracy: 0.8809\n",
      "Test acc: 0.881\n",
      "time: 49.5 s (started: 2021-08-04 16:49:24 +08:00)\n"
     ]
    }
   ],
   "source": [
    "model = keras.models.load_model(\"embeddings_bidir_gru_with_masking.keras\")\n",
    "print(f\"Test acc: {model.evaluate(int_test_ds)[1]:.3f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "clinical-frost",
   "metadata": {},
   "source": [
    "这一次，我们达到了 88% 的测试准确率——这是一个很小但很明显的改进。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "revolutionary-victorian",
   "metadata": {},
   "source": [
    "**使用预训练词嵌入**"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fossil-demonstration",
   "metadata": {},
   "source": [
    "有时，您可用的训练数据太少，以至于您无法单独使用数据来学习合适的特定于任务的词汇嵌入。 在这种情况下，您可以从预先计算的嵌入空间加载嵌入向量，而不是与您要解决的问题一起学习词嵌入，该嵌入空间您知道它是高度结构化的并具有有用的属性 - 捕获语言结构的通用方面。 在自然语言处理中使用预训练词嵌入背后的基本原理与在图像分类中使用预训练卷积网络非常相似：您没有足够的可用数据来自行学习真正强大的功能，但您期望获得所需的功能 相当通用——即常见的视觉特征或语义特征。 在这种情况下，重用在不同问题上学到的特征是有意义的。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "three-wallet",
   "metadata": {},
   "source": [
    "这种词嵌入通常是使用词出现统计（观察哪些词在句子或文档中共同出现）计算的，使用各种技术，有些涉及神经网络，有些则没有。 Bengio 等人最初探索了以无监督方式计算的密集、低维词嵌入空间的想法。 在 2000 年代初期，但在 24 种最著名和最成功的词嵌入方案之一：Word2Vec 算法 (code.google.com/archive/p/word2vec) 发布后，它才开始在研究和行业应用中起飞 ，由 Google 的 Tomas Mikolov 于 2013 年开发。 Word2Vec 维度捕获特定的语义属性，例如性别。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "split-nelson",
   "metadata": {},
   "source": [
    "您可以在 Keras 层下载和使用各种预先计算的词嵌入数据库。 Word2vec 就是其中之一。 另一种流行的方法称为词表示的全局嵌入向量（GloVe，nlp.stanford.edu/projects/glove），由斯坦福大学的研究人员于 2014 年开发。这种嵌入技术基于分解词共现统计矩阵。 它的开发人员已经为数百万个英语标记提供了预先计算的嵌入，这些标记是从维基百科数据和 Common Crawl 数据中获得的。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "preceding-fleece",
   "metadata": {},
   "source": [
    "让我们看看如何开始在 Keras 模型中使用 GloVe 嵌入。 同样的方法适用于 Word2Vec 嵌入或任何其他词嵌入数据库。 您还将使用此示例来刷新前几段介绍的文本标记化技术：您将从原始文本开始，然后逐步向上。\n",
    "\n",
    "下载 GloVe 词嵌入\n",
    "\n",
    "首先，让我们下载在 2014 年英文维基百科数据集上预先计算的 GloVe 词嵌入。 这是一个 822 MB 的 zip 文件，其中包含 400,000 个单词（或非单词标记）的 100 维嵌入向量。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "organic-prerequisite",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash\n",
    "\n",
    "cd data\n",
    "\n",
    "wget http://nlp.stanford.edu/data/glove.6B.zip && unzip -q glove.6B.zip"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "emerging-textbook",
   "metadata": {},
   "source": [
    "让我们解析解压后的文件（一个 .txt 文件）来构建一个索引，将单词（作为字符串）映射到它们的向量表示。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "played-eugene",
   "metadata": {},
   "source": [
    "> 清单 11.20 解析 GloVe word-embeddings 文件"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "legitimate-tunnel",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Found 400000 word vectors.\n",
      "time: 11.7 s (started: 2021-08-04 18:06:28 +08:00)\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "path_to_glove_file = \"data/glove.6B.100d.txt\"\n",
    "\n",
    "embeddings_index = {}\n",
    "\n",
    "with open(path_to_glove_file) as f:\n",
    "    for line in f:\n",
    "        word, coefs = line.split(maxsplit=1)\n",
    "        coefs = np.fromstring(coefs, \"f\", sep=\" \")\n",
    "        embeddings_index[word] = coefs\n",
    "\n",
    "print(f\"Found {len(embeddings_index)} word vectors.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "twenty-expression",
   "metadata": {},
   "source": [
    "**在模型中加载 GloVe 嵌入**\n",
    "\n",
    "接下来，让我们构建一个可以加载到层中的嵌入矩阵。 它必须是一个形状为 (max_words, embedding_dim) 的嵌入矩阵，其中每个条目 i 包含参考词索引中索引词的 embedding_dim 维向量（在标记化期间构建 i）。 请注意，索引 0 不应该代表任何单词或标记——它是一个占位符。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "painted-agreement",
   "metadata": {},
   "source": [
    "> 清单 11.21 准备 GloVe 词嵌入矩阵"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "electrical-swiss",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "time: 153 ms (started: 2021-08-04 18:08:56 +08:00)\n"
     ]
    }
   ],
   "source": [
    "embedding_dim = 100\n",
    "\n",
    "# 检索我们之前的 TextVectorization 层索引的词汇表\n",
    "vocabulary = text_vectorization.get_vocabulary()\n",
    "\n",
    "# 使用它来创建从单词到它们在词汇表中的索引的映射\n",
    "word_index = dict(zip(vocabulary, range(len(vocabulary)))) \n",
    "\n",
    "# 准备一个矩阵，我们将用 GloVe 向量填充\n",
    "embedding_matrix = np.zeros((max_tokens, embedding_dim)) \n",
    "for word, i in word_index.items():\n",
    "    if i < max_tokens:\n",
    "        embedding_vector = embeddings_index.get(word)\n",
    "        \n",
    "#     用索引 i 的词向量填充矩阵中的条目 i 。 在嵌入索引中未找到的单词将全部为零。\n",
    "    if embedding_vector is not None:\n",
    "        embedding_matrix[i] = embedding_vector"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "equal-retirement",
   "metadata": {},
   "source": [
    "最后，我们使用初始化器将预训练的嵌入加载到一个层中。 不断嵌入 为了在训练过程中不破坏预训练的表示，我们通过 trainable=False 冻结层"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "latin-football",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "time: 10.9 ms (started: 2021-08-04 18:08:59 +08:00)\n"
     ]
    }
   ],
   "source": [
    "embedding_layer = layers.Embedding(max_tokens,\n",
    "                                   embedding_dim,\n",
    "                                   embeddings_initializer=keras.initializers.Constant(embedding_matrix),\n",
    "                                   trainable=False,\n",
    "                                   mask_zero=True,\n",
    "                                  )"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "olympic-montgomery",
   "metadata": {},
   "source": [
    "在 GloVe 嵌入之上训练一个简单的双向 LSTM 我们现在准备训练一个新模型——与我们之前的模型相同，但利用 100 维预训练的 GloVe 嵌入而不是 128 维学习嵌入。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "perceived-truth",
   "metadata": {},
   "source": [
    "> 清单 11.22 使用预训练嵌入层的模型"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "instructional-mileage",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"model_3\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "input_4 (InputLayer)         [(None, None)]            0         \n",
      "_________________________________________________________________\n",
      "embedding_4 (Embedding)      (None, None, 100)         2000000   \n",
      "_________________________________________________________________\n",
      "bidirectional_3 (Bidirection (None, 64)                34048     \n",
      "_________________________________________________________________\n",
      "dropout_3 (Dropout)          (None, 64)                0         \n",
      "_________________________________________________________________\n",
      "dense_3 (Dense)              (None, 1)                 65        \n",
      "=================================================================\n",
      "Total params: 2,034,113\n",
      "Trainable params: 34,113\n",
      "Non-trainable params: 2,000,000\n",
      "_________________________________________________________________\n",
      "Epoch 1/10\n",
      "625/625 [==============================] - 92s 122ms/step - loss: 0.5806 - accuracy: 0.6931 - val_loss: 0.4537 - val_accuracy: 0.7912\n",
      "Epoch 2/10\n",
      "625/625 [==============================] - 70s 112ms/step - loss: 0.4567 - accuracy: 0.7897 - val_loss: 0.4158 - val_accuracy: 0.8106\n",
      "Epoch 3/10\n",
      "625/625 [==============================] - 74s 118ms/step - loss: 0.4029 - accuracy: 0.8232 - val_loss: 0.3769 - val_accuracy: 0.8350\n",
      "Epoch 4/10\n",
      "625/625 [==============================] - 72s 115ms/step - loss: 0.3684 - accuracy: 0.8411 - val_loss: 0.3718 - val_accuracy: 0.8340\n",
      "Epoch 5/10\n",
      "625/625 [==============================] - 71s 113ms/step - loss: 0.3467 - accuracy: 0.8518 - val_loss: 0.3290 - val_accuracy: 0.8586\n",
      "Epoch 6/10\n",
      "625/625 [==============================] - 72s 115ms/step - loss: 0.3264 - accuracy: 0.8615 - val_loss: 0.3303 - val_accuracy: 0.8598\n",
      "Epoch 7/10\n",
      "625/625 [==============================] - 71s 114ms/step - loss: 0.3104 - accuracy: 0.8712 - val_loss: 0.3030 - val_accuracy: 0.8710\n",
      "Epoch 8/10\n",
      "625/625 [==============================] - 75s 120ms/step - loss: 0.2948 - accuracy: 0.8783 - val_loss: 0.3173 - val_accuracy: 0.8688\n",
      "Epoch 9/10\n",
      "625/625 [==============================] - 72s 115ms/step - loss: 0.2790 - accuracy: 0.8872 - val_loss: 0.2888 - val_accuracy: 0.8808\n",
      "Epoch 10/10\n",
      "625/625 [==============================] - 73s 116ms/step - loss: 0.2670 - accuracy: 0.8909 - val_loss: 0.2877 - val_accuracy: 0.8788\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<tensorflow.python.keras.callbacks.History at 0x7f60c035b2b0>"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "time: 13min 7s (started: 2021-08-04 18:09:09 +08:00)\n"
     ]
    }
   ],
   "source": [
    "inputs = keras.Input(shape=(None,), dtype=\"int64\")\n",
    "\n",
    "embedded = embedding_layer(inputs)\n",
    "x = layers.Bidirectional(layers.LSTM(32))(embedded)\n",
    "x = layers.Dropout(0.5)(x)\n",
    "outputs = layers.Dense(1, activation=\"sigmoid\")(x)\n",
    "\n",
    "model = keras.Model(inputs, outputs)\n",
    "\n",
    "model.compile(optimizer=\"rmsprop\",\n",
    "              loss=\"binary_crossentropy\",\n",
    "              metrics=[\"accuracy\"])\n",
    "\n",
    "model.summary()\n",
    "\n",
    "callbacks = [keras.callbacks.ModelCheckpoint(\"glove_embeddings_sequence_model.keras\",\n",
    "                                             save_best_only=True)]\n",
    "\n",
    "model.fit(int_train_ds, validation_data=int_val_ds, epochs=10, callbacks=callbacks)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "expressed-watershed",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "782/782 [==============================] - 43s 48ms/step - loss: 0.2877 - accuracy: 0.8777\n",
      "Test acc: 0.878\n",
      "time: 49.3 s (started: 2021-08-04 18:22:16 +08:00)\n"
     ]
    }
   ],
   "source": [
    "model = keras.models.load_model(\"glove_embeddings_sequence_model.keras\")\n",
    "print(f\"Test acc: {model.evaluate(int_test_ds)[1]:.3f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "junior-medline",
   "metadata": {},
   "source": [
    "你会发现在这个特定的任务中，预训练的嵌入并不是很有帮助，因为数据集包含足够的样本，可以从头开始学习一个足够专业的嵌入空间。 但是，当您使用较小的数据集时，利用预训练嵌入可能会非常有用。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "mental-medline",
   "metadata": {},
   "source": [
    "## Transformer结构"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "formal-richmond",
   "metadata": {},
   "source": [
    "从 2017 年开始，一种新的模型架构开始在大多数自然语言处理任务中超越循环神经网络：Transformer。\n",
    "\n",
    "在 Vaswani 等人的开创性论文“Attention is all you need”中介绍了 Transformer。 论文的要点就在标题中：事实证明，一种称为“神经注意力”的简单机制可用于构建不具有任何循环层或卷积层的强大序列模型。\n",
    "\n",
    "这一发现在自然语言处理领域掀起了一场革命——甚至更远。 神经注意力已经迅速成为深度学习中最具影响力的思想之一。 在本节中，您将深入了解它的工作原理以及为什么它被证明对序列数据如此有效。 然后，您将利用自注意力创建一个 Transformer 编码器，这是 Transformer 架构的基本组件之一，并将其应用于 IMDB 电影评论分类任务。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "british-myanmar",
   "metadata": {},
   "source": [
    "### 理解自注意力"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "multiple-directive",
   "metadata": {},
   "source": [
    "在阅读本书时，您可能会略读某些部分并专心阅读其他部分，这取决于您的目标或兴趣是什么。 如果你的模型也这样做呢？ 这是一个简单而强大的想法：并非模型看到的所有输入信息对手头的任务都同等重要，因此模型应该“多关注”某些特征，而“少关注”其他特征。\n",
    "\n",
    "这听起来很熟悉吗？ 您已经在本书中两次遇到过类似的概念：\n",
    "* convnets 中的最大池化查看空间区域中的特征池，并仅选择一个特征来保留。 这是一种“全有或全无”的注意力形式：保留最重要的特征并丢弃其余特征。\n",
    "* TF-IDF 归一化根据不同令牌可能携带的信息量为令牌分配重要性分数。 重要的令牌得到提升，而不相关的令牌淡出。 这是一种持续的关注形式。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "sought-acquisition",
   "metadata": {},
   "source": [
    "你可以想象出许多不同形式的注意力，但它们都是从计算一组特征的重要性得分开始的，对更相关的特征得分较高，对不太相关的特征得分较低。 这些分数应该如何计算，以及你应该用它们做什么，会因方法而异。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ecological-vegetable",
   "metadata": {},
   "source": [
    "![](https://tva1.sinaimg.cn/large/008i3skNgy1gt4vjhrw2wj30v60u0tb6.jpg)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "operational-colorado",
   "metadata": {},
   "source": [
    "至关重要的是，这种注意力机制不仅仅可以用于突出或擦除某些特征。 它可用于使功能具有上下文感知能力。 您刚刚了解了词嵌入——捕捉不同词之间语义关系“形状”的向量空间。 在嵌入空间中，单个单词具有固定位置——与空间中其他单词的一组固定关系。 但这并不是语言的运作方式：一个词的含义通常是特定于上下文的。 当您标记日期时，您不是在谈论与约会时相同的“日期”，也不是您在市场上购买的那种日期。 当你说“我很快就会见到你”时，“看到”这个词的含义与“我会看到这个项目结束”或“我明白你的意思”中的“看到”有着微妙的不同 . 当然，“他”、“它”、“在”等代词的含义完全是特定于句子的，甚至可以在一个句子中多次变化。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "military-sheriff",
   "metadata": {},
   "source": [
    "显然，智能嵌入空间将为一个词提供不同的向量表示，具体取决于它周围的其他词。 这就是 self-attention 的用武之地。 self-attention 的目的是通过使用序列中相关标记的表示来调整标记的表示。 这会产生上下文感知标记表示。 考虑一个例句：“火车准时离开了车站。” 现在，考虑句子中的一个词：“站”。 我们在谈论什么样的车站？ 会不会是电台？ 也许是国际空间站？ 让我们通过 self-attention 在算法上计算出来（见图 11.6）。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cooperative-combination",
   "metadata": {},
   "source": [
    "![](https://tva1.sinaimg.cn/large/008i3skNgy1gt4vm1uq6yj31b20tw438.jpg)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "distinct-retreat",
   "metadata": {},
   "source": [
    "第 1 步：计算“station”的向量与句子中所有其他单词之间的相关性分数。这些是我们的“注意力分数”。我们将简单地使用两个词向量之间的点积来衡量它们之间的关系强度。这是一个计算效率非常高的距离函数，早在 Transformers 之前，它就已经是将两个词嵌入相互关联的标准方法。在实践中，这些分数也会经过一个缩放函数和一个 softmax，但现在，这只是一个实现细节。\n",
    "\n",
    "第 2 步：计算句子中所有词向量的总和，由我们的相关性得分加权。与“station”密切相关的词对总和的贡献更大（包括“station”一词\n",
    "本身），而无关的词几乎没有任何贡献。由此产生的向量是我们对“站”的新表示：一种结合了周围环境的表示。特别是，它包含了“火车”向量的一部分，说明它实际上是一个“火车站”。\n",
    "\n",
    "你会对句子中的每个词重复这个过程，产生一个新的编码句子的向量序列。让我们在类似 NumPy 的伪代码中看到它："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "id": "earned-disco",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "time: 11.6 ms (started: 2021-08-04 18:27:14 +08:00)\n"
     ]
    }
   ],
   "source": [
    "def self_attention(input_sequence):\n",
    "    \n",
    "    output = np.zeros(shape=input_sequence.shape)\n",
    "    \n",
    "#     迭代输入序列中的每个标记\n",
    "    for i, pivot_vector in enumerate(input_sequence): \n",
    "        scores = np.zeros(shape=(len(input_sequence),))\n",
    "        \n",
    "        for j, vector in enumerate(input_sequence):\n",
    "#             计算令牌和其他所有令牌之间的点积（注意力分数）\n",
    "            scores[j] = np.dot(pivot_vector, vector.T) \n",
    "    \n",
    "#         按归一化因子缩放并应用 softmax\n",
    "        scores /= np.sqrt(input_sequence.shape[1]) \n",
    "        scores = softmax(scores)\n",
    "        \n",
    "        \n",
    "        new_pivot_representation = np.zeros(shape=pivot_vector.shape)\n",
    "        for j, vector in enumerate(input_sequence):\n",
    "#             取注意力分数加权的所有标记的总和\n",
    "            new_pivot_representation += vector * scores[j] \n",
    "    \n",
    "#         那个总和就是我们的输出\n",
    "        output[i] = new_pivot_representation \n",
    "    return output"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "broken-pulse",
   "metadata": {},
   "source": [
    "当然，在实践中您会使用矢量化实现。 Keras 有一个内置层来处理它：MultiHeadAttention 层。 以下是您将如何使用它："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "finite-lounge",
   "metadata": {},
   "outputs": [],
   "source": [
    "num_heads = 4\n",
    "embed_dim = 256\n",
    "\n",
    "mha_layer = MultiHeadAttention(num_heads=num_heads, key_dim=embed_dim)\n",
    "\n",
    "outputs = mha_layer(inputs, inputs, inputs)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "studied-suicide",
   "metadata": {},
   "source": [
    "读到这里，你可能想知道：\n",
    "为什么将输入传递给层三次？ 这似乎是多余的。 这些“多头”指的是什么？ 这听起来很吓人——如果你剪掉它们，它们还会长出来吗？\n",
    "\n",
    "这两个问题都有简单的答案。 让我们来看看。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "interim-package",
   "metadata": {},
   "source": [
    "**GENERALIZED SELF-ATTENTION：查询键值模型**"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "thick-sleep",
   "metadata": {},
   "source": [
    "到目前为止，我们只考虑了一个输入序列。 然而，Transformer 架构最初是为机器翻译而开发的，您必须在其中处理两个输入序列：您当前正在翻译的源序列（例如“今天天气如何？”），以及您正在转换的目标序列 它到（例如“¿Qué tiempo hace hoy？”）。 Transformer 是一种序列到序列模型：它旨在将一个序列转换为另一个序列。 您将在本章后面深入了解序列到序列模型。\n",
    "\n",
    "现在，让我们退后一步。 我们介绍的自注意力机制执行以下操作，示意性地："
   ]
  },
  {
   "cell_type": "markdown",
   "id": "palestinian-viewer",
   "metadata": {},
   "source": [
    "![](https://tva1.sinaimg.cn/large/008i3skNgy1gt4xtr7zjhj317u07iq3c.jpg)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "sporting-greeting",
   "metadata": {},
   "source": [
    "这意味着：“对于输入（A）中的每个标记，计算该标记与输入（B）中的每个标记的相关程度，然后使用这些分数对输入的标记总和进行加权”。至关重要的是，没有什么需要 A 、B 和 C 指代相同的输入序列。 在一般情况下，您可以使用三个不同的序列来执行此操作。 我们将它们称为“查询”、“键”和“值”。 操作变成：“对于查询中的每个元素，计算元素与每个键的相关程度，这些使用这些分数来加权值的总和”。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "recorded-adjustment",
   "metadata": {},
   "source": [
    "![](https://tva1.sinaimg.cn/large/008i3skNgy1gt4xww1sbfj318o04a3yr.jpg)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "responsible-vacation",
   "metadata": {},
   "source": [
    "该术语来自搜索引擎和推荐系统。想象一下，您正在输入一个查询以从您的收藏中检索一张照片——“海滩上的狗”。在内部，数据库中的每张图片都由一组关键字描述——“猫”、“狗”、“派对”等。我们将这些称为“键”。搜索引擎将首先将您的查询与数据库中的键进行比较。 “Dog” 产生 1 的匹配，“cat”产生 0 的匹配。然后它会根据匹配强度（相关性）对这些键进行排名，并按照相关性的顺序返回与前 N 个匹配相关联的图片。\n",
    "\n",
    "从概念上讲，这就是 Transformer 风格的注意力正在做的事情。你有一个描述你正在寻找的东西的参考序列：查询。你有一个你试图从中提取信息的知识体系：价值观。每个值都分配了一个键，该键以易于与查询进行比较的格式描述该值。您只需将查询与键匹配即可。然后返回值的加权总和。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fitted-absorption",
   "metadata": {},
   "source": [
    "![](https://tva1.sinaimg.cn/large/008i3skNgy1gt4xzwrbuoj30wg0k6jsy.jpg)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "viral-dressing",
   "metadata": {},
   "source": [
    "实际上，键和值通常是相同的序列。 例如，在机器翻译中，查询将是目标序列，而源序列将扮演键和值的角色：对于目标的每个元素（如“tiempo”），您想回到源 （“今天的天气怎么样？”）并确定与之相关的不同位（“tiempo”和“weather”应该有很强的匹配）。 很自然地，如果你只是在做序列分类，那么查询、键和值都是一样的：你将一个序列与其自身进行比较，用整个序列的上下文来丰富每个标记。\n",
    "\n",
    "这就解释了为什么我们需要将输入三次传递给 MultiHeadAttention 层。 但为什么要“多头”关注？"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "adequate-stick",
   "metadata": {},
   "source": [
    "### 多头注意力"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "arbitrary-ancient",
   "metadata": {},
   "source": [
    "“多头注意力”是对自注意力机制的额外调整，在“注意力就是你所需要的”中介绍。 “多头”绰号是指自注意力层的输出空间被分解为一组独立的子空间，分别学习：初始查询、键和值，并通过三个独立的组发送 密集投影，产生三组调制查询、键和值，每组都通过神经网络处理\n",
    "注意，三个输出连接在一起成为一个输出序列。每个这样的子空间称为“头”。\n",
    "\n",
    "完整的图片是这样的："
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cognitive-blend",
   "metadata": {},
   "source": [
    "![](https://tva1.sinaimg.cn/large/008i3skNgy1gt4y4wrxikj317b0u0q5h.jpg)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bibliographic-despite",
   "metadata": {},
   "source": [
    "可学习的密集投影的存在使该层能够实际学习一些东西，而不是纯粹的无状态转换，需要在它之前或之后附加层才能有用。此外，拥有独立的头有助于该层为每个令牌学习不同的特征组，其中一组内的特征彼此相关，但大部分独立于不同组中的特征。\n",
    "\n",
    "这在原理上类似于使深度可分离卷积起作用的原理：在深度可分离卷积中，卷积的输出空间被分解为许多独立学习的子空间（每个输入通道一个）。论文“Attention is all you need”是在将特征空间分解为独立子空间的想法已被证明为计算机视觉模型提供了巨大好处的时候写的——无论是在深度可分离卷积的情况下，还是在这种情况下一种密切相关的方法，分组卷积。多头注意力只是将相同的想法应用于自注意力。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "angry-cincinnati",
   "metadata": {},
   "source": [
    "### Transformer编码器"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "civil-monitor",
   "metadata": {},
   "source": [
    "如果添加额外的密集投影如此有用，为什么我们不也将一两个应用于注意力机制的输出？实际上，这是个好主意，让我们这样做。你知道吗，我们的模型开始做很多事情，所以我们可能想要添加残差连接，以确保我们不会在此过程中破坏任何有价值的信息——我们在第 9 章中了解到它们是对于任何足够深的架构来说都是必须的。哦，还有我们在第 9 章中学到的另一件事：归一化层应该帮助梯度更好地流动\n",
    "反向传播。让我们也添加这些。\n",
    "\n",
    "以上大致是我想象中在当时 Transformer 架构的发明者脑海中展开的思考过程。将输出分解为多个独立空间、添加残差连接、添加归一化层——所有这些都是标准架构模式，在任何复杂模型中都可以明智地加以利用。这些花里胡哨的元素共同构成了 Transformer 编码器——构成 Transformer 架构的两个关键部分之一。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "graphic-manor",
   "metadata": {},
   "source": [
    "![](https://tva1.sinaimg.cn/large/008i3skNgy1gt4ycqkuiij30kg0qe3zs.jpg)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "comic-investor",
   "metadata": {},
   "source": [
    "原始的 Transformer 架构由两部分组成：处理源序列的 Transformer 编码器和使用源序列生成翻译版本的 Transformer 解码器。 您将在一分钟内了解解码器部分。\n",
    "\n",
    "至关重要的是，编码器部分可用于文本分类——它是一个非常通用的模块，它摄取序列并学习将其转换为更有用的表示。 让我们实现一个 Transformer 编码器，并在电影评论情感分类任务中试用它。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "chicken-wireless",
   "metadata": {},
   "source": [
    "> 清单 11.23 作为子类层实现的 Transformer 编码器"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "difficult-booth",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "time: 4.93 ms (started: 2021-08-05 12:43:13 +08:00)\n"
     ]
    }
   ],
   "source": [
    "import tensorflow as tf\n",
    "from tensorflow import keras\n",
    "from tensorflow.keras import layers\n",
    "\n",
    "class TransformerEncoder(layers.Layer):\n",
    "    def __init__(self, embed_dim, dense_dim, num_heads, **kwargs):\n",
    "        super().__init__(**kwargs)\n",
    "        \n",
    "        self.embed_dim = embed_dim # 输入标记向量的大小\n",
    "        self.dense_dim = dense_dim # 内密层尺寸\n",
    "        self.num_heads = num_heads # 注意力头数\n",
    "        \n",
    "        self.attention = layers.MultiHeadAttention(num_heads=num_heads, \n",
    "                                                   key_dim=embed_dim)\n",
    "        self.dense_proj = keras.Sequential([layers.Dense(dense_dim, activation=\"relu\"),\n",
    "                                            layers.Dense(embed_dim),])\n",
    "        self.layernorm_1 = layers.LayerNormalization()\n",
    "        self.layernorm_2 = layers.LayerNormalization()\n",
    "        \n",
    "    def call(self, inputs, mask=None): # 计算在call()中\n",
    "        # 小细节：Embedding 层将生成的掩码将是 2D 的，但注意力层预计是 3D 或 4D，因此我们扩大了它的等级。\n",
    "        if mask is not None:\n",
    "            mask = mask[:, tf.newaxis, :] \n",
    "        attention_output = self.attention(inputs, inputs, attention_mask=mask)\n",
    "        proj_input = self.layernorm_1(inputs + attention_output)\n",
    "        proj_output = self.dense_proj(proj_input)\n",
    "        return self.layernorm_2(proj_input + proj_output)\n",
    "    \n",
    "    def get_config(self): # 实现序列化以便我们可以保存模型\n",
    "        config = super().get_config()\n",
    "        config.update({\"embed_dim\": self.embed_dim, \n",
    "                       \"num_heads\": self.num_heads,\n",
    "                       \"dense_dim\": self.dense_dim,\n",
    "                      })\n",
    "        return config"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "higher-vault",
   "metadata": {},
   "source": [
    "您会注意到，我们在这里使用的标准化层与您之前在图像模型中使用过的 BatchNormalization 层不同。 这是因为 BatchNormalization 不适用于序列数据。 相反，我们使用 LayerNormalization 层，它独立于批次中的其他序列对每个序列进行标准化。 像这样，在类似 NumPy 的伪代码中："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "id": "loaded-purse",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "time: 1.18 ms (started: 2021-08-04 19:14:54 +08:00)\n"
     ]
    }
   ],
   "source": [
    "def layer_normalization(batch_of_sequences): #输入形状：（batch_size、sequence_length、embedding_dim）。\n",
    "    # 为了计算均值和方差，我们只在最后一个轴（轴 -1）上合并数据。\n",
    "    mean = np.mean(batch_of_sequences, keepdims=True, axis=-1) \n",
    "    variance = np.var(batch_of_sequences, keepdims=True, axis=-1) \n",
    "    \n",
    "    return (batch_of_sequences - mean) / variance"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "worthy-direction",
   "metadata": {},
   "source": [
    "与 BatchNormalization 比较（在训练期间）："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "id": "floral-preparation",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "time: 1.42 ms (started: 2021-08-04 19:31:20 +08:00)\n"
     ]
    }
   ],
   "source": [
    "def batch_normalization(batch_of_images): # 输入形状：（batch_size、高度、宽度、通道）。\n",
    "#     在批次轴（轴 0）上汇集数据，这会在批次中创建样本之间的相互作用。\n",
    "    mean = np.mean(batch_of_images, keepdims=True, axis=(0, 1, 2)) \n",
    "    variance = np.var(batch_of_images, keepdims=True, axis=(0, 1, 2)) \n",
    "    return (batch_of_images - mean) / variance"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "phantom-focus",
   "metadata": {},
   "source": [
    "BatchNormalization 从多个样本中收集信息以获得特征均值和方差的准确统计数据，而 LayerNormalization 仅将每个序列内的数据单独池化，更适合序列数据。\n",
    "\n",
    "现在我们已经实现了我们的 TransformerEncoder，我们可以使用它来组装一个类似于您之前看到的基于 GRU 的文本分类模型："
   ]
  },
  {
   "cell_type": "markdown",
   "id": "union-playback",
   "metadata": {},
   "source": [
    "> 清单 11.24 结合了 Transformer 编码器和池化层的文本分类模型"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "id": "assured-award",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"model_4\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "input_5 (InputLayer)         [(None, None)]            0         \n",
      "_________________________________________________________________\n",
      "embedding_5 (Embedding)      (None, None, 256)         5120000   \n",
      "_________________________________________________________________\n",
      "transformer_encoder (Transfo (None, None, 256)         543776    \n",
      "_________________________________________________________________\n",
      "global_max_pooling1d (Global (None, 256)               0         \n",
      "_________________________________________________________________\n",
      "dropout_4 (Dropout)          (None, 256)               0         \n",
      "_________________________________________________________________\n",
      "dense_6 (Dense)              (None, 1)                 257       \n",
      "=================================================================\n",
      "Total params: 5,664,033\n",
      "Trainable params: 5,664,033\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n",
      "time: 713 ms (started: 2021-08-04 19:37:49 +08:00)\n"
     ]
    }
   ],
   "source": [
    "vocab_size = 20000\n",
    "embed_dim = 256\n",
    "num_heads = 2\n",
    "dense_dim = 32\n",
    "\n",
    "inputs = keras.Input(shape=(None,), dtype=\"int64\")\n",
    "x = layers.Embedding(vocab_size, embed_dim)(inputs)\n",
    "x = TransformerEncoder(embed_dim, dense_dim, num_heads)(x)\n",
    "\n",
    "# 由于 TransformerEncoder 返回完整序列，我们需要通过全局池化层将每个序列减少为单个向量以进行分类。\n",
    "x = layers.GlobalMaxPooling1D()(x) \n",
    "x = layers.Dropout(0.5)(x)\n",
    "outputs = layers.Dense(1, activation=\"sigmoid\")(x)\n",
    "\n",
    "model = keras.Model(inputs, outputs)\n",
    "\n",
    "model.compile(optimizer=\"rmsprop\",\n",
    "              loss=\"binary_crossentropy\",\n",
    "              metrics=[\"accuracy\"])\n",
    "\n",
    "model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "colored-environment",
   "metadata": {},
   "source": [
    "让我们训练它。 它达到了 87.5% 的测试准确率——比 GRU 模型略差。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "rotary-austria",
   "metadata": {},
   "source": [
    "清单 11.25 训练和评估基于 Transformer 编码器的模型"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "id": "sporting-blood",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/20\n",
      "625/625 [==============================] - 77s 117ms/step - loss: 0.5015 - accuracy: 0.7721 - val_loss: 0.3333 - val_accuracy: 0.8546\n",
      "Epoch 2/20\n",
      "625/625 [==============================] - 74s 118ms/step - loss: 0.3277 - accuracy: 0.8617 - val_loss: 0.2899 - val_accuracy: 0.8800\n",
      "Epoch 3/20\n",
      "625/625 [==============================] - 76s 121ms/step - loss: 0.2651 - accuracy: 0.8909 - val_loss: 0.2742 - val_accuracy: 0.8928\n",
      "Epoch 4/20\n",
      "625/625 [==============================] - 77s 123ms/step - loss: 0.2048 - accuracy: 0.9176 - val_loss: 0.3323 - val_accuracy: 0.8748\n",
      "Epoch 5/20\n",
      "625/625 [==============================] - 76s 121ms/step - loss: 0.1638 - accuracy: 0.9373 - val_loss: 0.3529 - val_accuracy: 0.8766\n",
      "Epoch 6/20\n",
      "625/625 [==============================] - 72s 115ms/step - loss: 0.1378 - accuracy: 0.9488 - val_loss: 0.4422 - val_accuracy: 0.8686\n",
      "Epoch 7/20\n",
      "625/625 [==============================] - 75s 120ms/step - loss: 0.1193 - accuracy: 0.9556 - val_loss: 0.4410 - val_accuracy: 0.8746\n",
      "Epoch 8/20\n",
      "625/625 [==============================] - 73s 116ms/step - loss: 0.1015 - accuracy: 0.9616 - val_loss: 0.5797 - val_accuracy: 0.8730\n",
      "Epoch 9/20\n",
      "625/625 [==============================] - 74s 119ms/step - loss: 0.0912 - accuracy: 0.9669 - val_loss: 0.5480 - val_accuracy: 0.8688\n",
      "Epoch 10/20\n",
      "625/625 [==============================] - 76s 122ms/step - loss: 0.0796 - accuracy: 0.9721 - val_loss: 0.5476 - val_accuracy: 0.8556\n",
      "Epoch 11/20\n",
      "625/625 [==============================] - 71s 113ms/step - loss: 0.0716 - accuracy: 0.9743 - val_loss: 0.5534 - val_accuracy: 0.8676\n",
      "Epoch 12/20\n",
      "625/625 [==============================] - 75s 119ms/step - loss: 0.0642 - accuracy: 0.9768 - val_loss: 0.5729 - val_accuracy: 0.8656\n",
      "Epoch 13/20\n",
      "625/625 [==============================] - 77s 123ms/step - loss: 0.0575 - accuracy: 0.9801 - val_loss: 0.6590 - val_accuracy: 0.8614\n",
      "Epoch 14/20\n",
      "625/625 [==============================] - 74s 119ms/step - loss: 0.0498 - accuracy: 0.9822 - val_loss: 0.6717 - val_accuracy: 0.8630\n",
      "Epoch 15/20\n",
      "625/625 [==============================] - 77s 123ms/step - loss: 0.0420 - accuracy: 0.9865 - val_loss: 0.6505 - val_accuracy: 0.8658\n",
      "Epoch 16/20\n",
      "625/625 [==============================] - 72s 116ms/step - loss: 0.0355 - accuracy: 0.9872 - val_loss: 0.8028 - val_accuracy: 0.8468\n",
      "Epoch 17/20\n",
      "625/625 [==============================] - 74s 119ms/step - loss: 0.0346 - accuracy: 0.9890 - val_loss: 0.8030 - val_accuracy: 0.8654\n",
      "Epoch 18/20\n",
      "625/625 [==============================] - 74s 118ms/step - loss: 0.0278 - accuracy: 0.9912 - val_loss: 0.8728 - val_accuracy: 0.8604\n",
      "Epoch 19/20\n",
      "625/625 [==============================] - 77s 123ms/step - loss: 0.0252 - accuracy: 0.9920 - val_loss: 0.8838 - val_accuracy: 0.8666\n",
      "Epoch 20/20\n",
      " 53/625 [=>............................] - ETA: 1:06 - loss: 0.0127 - accuracy: 0.9947time: 23min 54s (started: 2021-08-04 19:46:38 +08:00)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n",
      "KeyboardInterrupt\n",
      "\n"
     ]
    }
   ],
   "source": [
    "callbacks = [keras.callbacks.ModelCheckpoint(\"transformer_encoder.keras\",\n",
    "                                             save_best_only=True)]\n",
    "\n",
    "model.fit(int_train_ds, validation_data=int_val_ds, epochs=20, callbacks=callbacks)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "timely-gamma",
   "metadata": {},
   "outputs": [],
   "source": [
    "model = keras.models.load_model(\"transformer_encoder.keras\",\n",
    "                                custom_objects={\"TransformerEncoder\": TransformerEncoder}) \n",
    "print(f\"Test acc: {model.evaluate(int_test_ds)[1]:.3f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "compact-academy",
   "metadata": {},
   "source": [
    "此时，您应该开始感到有些不安。这里有些不对劲。你能说出它是什么吗？\n",
    "\n",
    "本节表面上是关于“序列模型”的。我首先强调了词序的重要性。我说过 Transformer 是一种序列处理架构，最初是为机器翻译而开发的。然而……您刚刚看到的 Transformer 编码器根本不是序列模型。你注意到了吗？它由彼此独立处理序列标记的密集层和将标记视为一个集合的注意力层组成。你可以改变一个序列中标记的顺序，你会得到完全相同的成对注意力分数和完全相同的上下文感知表示。如果你在每篇电影评论中完全打乱单词，模型不会注意到，你仍然会得到完全相同的准确度。 Self-attention 是一种集合处理机制，专注于序列元素对之间的关​​系（见图 11.10）——它不知道这些元素是出现在序列的开头、结尾还是中间。等等，那为什么我们说Transformer是一个序列模型呢？如果它不看词序，它怎么可能对机器翻译有好处？"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "passing-landscape",
   "metadata": {},
   "source": [
    "我在本章前面暗示了解决方案：我顺便提到 Transformer 是一种混合方法，它在技术上与顺序无关，但它会在它处理的表示中手动注入顺序信息。 这是缺少的成分！ 它被称为“位置编码”。 让我们来看看。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "involved-deployment",
   "metadata": {},
   "source": [
    "![](https://tva1.sinaimg.cn/large/008i3skNgy1gt4zzp1oubj316y0to41c.jpg)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dimensional-finnish",
   "metadata": {},
   "source": [
    "**使用位置编码重新输入订单信息**"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "white-saint",
   "metadata": {},
   "source": [
    "位置编码背后的想法非常简单：为了让模型访问词序信息，我们将添加到每个词嵌入单词在句子中的位置。我们的输入词嵌入将有两个组成部分：通常的词向量，它表示独立于任何特定上下文的单词，以及一个位置向量，表示单词在当前句子中的位置。希望该模型随后会弄清楚如何最好地利用这些附加信息。\n",
    "\n",
    "你能想到的最简单的方案是将单词的位置连接到它的嵌入向量。您将向向量添加一个“位置”轴，并为序列中的第一个单词填充 0，为第二个单词填充 1，依此类推。\n",
    "\n",
    "然而，这可能并不理想，因为您的位置可能是非常大的整数，这会破坏嵌入向量中的值范围。如您所知，神经网络不喜欢非常大的输入值或离散输入分布。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "electronic-final",
   "metadata": {},
   "source": [
    "最初的“Attention is all you need paper”使用了一个有趣的技巧来编码词位置：它向词嵌入添加了一个向量，该向量包含范围内的值，该范围内的值根据位置循环变化 [-1, 1]（它使用余弦函数 为了达成这个）。 这个技巧提供了一种通过小值向量在大范围内唯一表征任何整数的方法。 这很聪明，但它不是我们将在我们的案例中使用的。 我们将做一些更简单、更有效的事情：我们将学习位置嵌入向量，就像我们学习嵌入单词索引一样。 然后我们将继续将我们的位置嵌入添加到相应的词嵌入中，以获得位置感知词嵌入。 这种技术称为“位置嵌入”。 让我们实现它："
   ]
  },
  {
   "cell_type": "markdown",
   "id": "reduced-maple",
   "metadata": {},
   "source": [
    "> 清单 11.26 将位置嵌入实现为子类层"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "coral-hollywood",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "time: 1.49 ms (started: 2021-08-05 12:42:49 +08:00)\n"
     ]
    }
   ],
   "source": [
    "class PositionalEmbedding(layers.Layer):\n",
    "#     位置嵌入的一个缺点是需要提前知道序列长度。\n",
    "    def __init__(self, sequence_length, input_dim, output_dim, **kwargs): \n",
    "        super().__init__(**kwargs)\n",
    "#         为令牌索引准备嵌入层\n",
    "        self.token_embeddings = layers.Embedding(input_dim=input_dim, output_dim=output_dim)\n",
    "#         另一个嵌入层用于令牌位置\n",
    "        self.position_embeddings = layers.Embedding(input_dim=sequence_length, \n",
    "                                                    output_dim=output_dim) \n",
    "        self.sequence_length = sequence_length\n",
    "        self.input_dim = input_dim\n",
    "        self.output_dim = output_dim\n",
    "        \n",
    "    def call(self, inputs):\n",
    "        length = tf.shape(inputs)[-1]\n",
    "        positions = tf.range(start=0, limit=length, delta=1)\n",
    "        embedded_tokens = self.token_embeddings(inputs)\n",
    "        embedded_positions = self.position_embeddings(positions)\n",
    "#         将两个嵌入向量相加\n",
    "        return embedded_tokens + embedded_positions\n",
    "    \n",
    "    \n",
    "#     与嵌入层一样，该层应该能够生成掩码，以便我们可以忽略输入中的填充 0。 \n",
    "#     框架将自动调用计算掩码方法，掩码将传播到下一层。\n",
    "    def compute_mask(self, inputs, mask=None): \n",
    "        return tf.math.not_equal(inputs, 0)\n",
    "    \n",
    "#     实现序列化以便我们可以保存模型\n",
    "    def get_config(self):\n",
    "        config = super().get_config()\n",
    "        config.update({\"output_dim\": self.output_dim,\n",
    "                       \"sequence_length\": self.sequence_length,\n",
    "                       \"input_dim\": self.input_dim,\n",
    "                      })\n",
    "        return config"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "respected-telling",
   "metadata": {},
   "source": [
    "您可以像使用常规嵌入层一样使用此 PositionEmbedding 层。 让我们看看它的实际效果！"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "elder-instruction",
   "metadata": {},
   "source": [
    "**保存自定义图层的注意事项** "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "upset-market",
   "metadata": {},
   "source": [
    "当您编写自定义层时，请确保实现 get_config 方法：这使层能够从其 config dict 重新实例化，这在模型保存和加载期间非常有用。 该方法应返回一个 Python 字典，其中包含用于创建层的构造函数参数的值。\n",
    "\n",
    "所有 Keras 层都可以通过以下方式序列化和反序列化："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "amber-hollywood",
   "metadata": {},
   "outputs": [],
   "source": [
    "config = layer.get_config()\n",
    "# 请注意，配置不包含权重值，因此层中的所有权重都从头开始初始化。\n",
    "new_layer = layer.__class__.from_config(config)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "worthy-mississippi",
   "metadata": {},
   "source": [
    "例如："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "found-rhythm",
   "metadata": {},
   "outputs": [],
   "source": [
    "layer = PositionalEmbedding(sequence_length, input_dim, output_dim)\n",
    "config = layer.get_config()\n",
    "new_layer = PositionalEmbedding.from_config(config)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "confirmed-encyclopedia",
   "metadata": {},
   "source": [
    "保存包含自定义层的模型时，保存文件将包含这些配置字典。 从文件加载模型时，您应该为加载过程提供自定义层类，以便它可以理解配置对象："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "sunset-bronze",
   "metadata": {},
   "outputs": [],
   "source": [
    "model = keras.models.load_model(filename, custom_objects={\"PositionalEmbedding\": PositionalEmbedding})"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "personal-croatia",
   "metadata": {},
   "source": [
    "**综合起来：文本分类转换器**"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "expressed-studio",
   "metadata": {},
   "source": [
    "要开始考虑词序，您所要做的就是将旧层与嵌入我们的位置感知版本交换："
   ]
  },
  {
   "cell_type": "markdown",
   "id": "august-soccer",
   "metadata": {},
   "source": [
    "> 清单 11.27 结合了位置嵌入、Transformer 编码器和池化层的文本分类模型"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "id": "graphic-brook",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"model_5\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "input_6 (InputLayer)         [(None, None)]            0         \n",
      "_________________________________________________________________\n",
      "positional_embedding (Positi (None, None, 256)         5273600   \n",
      "_________________________________________________________________\n",
      "transformer_encoder_1 (Trans (None, None, 256)         543776    \n",
      "_________________________________________________________________\n",
      "global_max_pooling1d_1 (Glob (None, 256)               0         \n",
      "_________________________________________________________________\n",
      "dropout_5 (Dropout)          (None, 256)               0         \n",
      "_________________________________________________________________\n",
      "dense_9 (Dense)              (None, 1)                 257       \n",
      "=================================================================\n",
      "Total params: 5,817,633\n",
      "Trainable params: 5,817,633\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n",
      "Epoch 1/20\n",
      "625/625 [==============================] - 79s 121ms/step - loss: 0.4778 - accuracy: 0.7883 - val_loss: 0.3059 - val_accuracy: 0.8774\n",
      "Epoch 2/20\n",
      "625/625 [==============================] - 79s 126ms/step - loss: 0.2348 - accuracy: 0.9074 - val_loss: 0.2910 - val_accuracy: 0.8972\n",
      "Epoch 3/20\n",
      "625/625 [==============================] - 77s 123ms/step - loss: 0.1773 - accuracy: 0.9319 - val_loss: 0.4351 - val_accuracy: 0.8682\n",
      "Epoch 4/20\n",
      "625/625 [==============================] - 79s 127ms/step - loss: 0.1416 - accuracy: 0.9478 - val_loss: 0.3369 - val_accuracy: 0.8860\n",
      "Epoch 5/20\n",
      "625/625 [==============================] - 76s 122ms/step - loss: 0.1223 - accuracy: 0.9566 - val_loss: 0.4792 - val_accuracy: 0.8822\n",
      "Epoch 6/20\n",
      "625/625 [==============================] - 79s 126ms/step - loss: 0.1071 - accuracy: 0.9632 - val_loss: 0.4051 - val_accuracy: 0.8816\n",
      "Epoch 7/20\n",
      "625/625 [==============================] - 75s 120ms/step - loss: 0.0950 - accuracy: 0.9683 - val_loss: 0.4732 - val_accuracy: 0.8802\n",
      "Epoch 8/20\n",
      "625/625 [==============================] - 79s 126ms/step - loss: 0.0886 - accuracy: 0.9697 - val_loss: 0.4932 - val_accuracy: 0.8832\n",
      "Epoch 9/20\n",
      "625/625 [==============================] - 76s 121ms/step - loss: 0.0747 - accuracy: 0.9758 - val_loss: 0.5849 - val_accuracy: 0.8782\n",
      "Epoch 10/20\n",
      "625/625 [==============================] - 76s 122ms/step - loss: 0.0713 - accuracy: 0.9765 - val_loss: 0.5376 - val_accuracy: 0.8766\n",
      "Epoch 11/20\n",
      "625/625 [==============================] - 78s 125ms/step - loss: 0.0668 - accuracy: 0.9782 - val_loss: 0.5141 - val_accuracy: 0.8776\n",
      "Epoch 12/20\n",
      "625/625 [==============================] - 80s 128ms/step - loss: 0.0595 - accuracy: 0.9794 - val_loss: 0.5217 - val_accuracy: 0.8744\n",
      "Epoch 13/20\n",
      "625/625 [==============================] - 75s 121ms/step - loss: 0.0560 - accuracy: 0.9809 - val_loss: 0.5026 - val_accuracy: 0.8738\n",
      "Epoch 14/20\n",
      "625/625 [==============================] - 77s 123ms/step - loss: 0.0508 - accuracy: 0.9837 - val_loss: 0.4964 - val_accuracy: 0.8790\n",
      "Epoch 15/20\n",
      "479/625 [=====================>........] - ETA: 16s - loss: 0.0438 - accuracy: 0.9864"
     ]
    },
    {
     "ename": "KeyboardInterrupt",
     "evalue": "",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mKeyboardInterrupt\u001b[0m                         Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-44-f8547fa44d28>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[1;32m     23\u001b[0m                                              save_best_only=True)]\n\u001b[1;32m     24\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 25\u001b[0;31m \u001b[0mmodel\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mfit\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mint_train_ds\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mvalidation_data\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mint_val_ds\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mepochs\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;36m20\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcallbacks\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mcallbacks\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[0;32m/public/huangwei/miniconda3/envs/tensorflow/lib/python3.8/site-packages/tensorflow/python/keras/engine/training.py\u001b[0m in \u001b[0;36mfit\u001b[0;34m(self, x, y, batch_size, epochs, verbose, callbacks, validation_split, validation_data, shuffle, class_weight, sample_weight, initial_epoch, steps_per_epoch, validation_steps, validation_batch_size, validation_freq, max_queue_size, workers, use_multiprocessing)\u001b[0m\n\u001b[1;32m   1181\u001b[0m                 _r=1):\n\u001b[1;32m   1182\u001b[0m               \u001b[0mcallbacks\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mon_train_batch_begin\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mstep\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1183\u001b[0;31m               \u001b[0mtmp_logs\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mtrain_function\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0miterator\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1184\u001b[0m               \u001b[0;32mif\u001b[0m \u001b[0mdata_handler\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mshould_sync\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1185\u001b[0m                 \u001b[0mcontext\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0masync_wait\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/public/huangwei/miniconda3/envs/tensorflow/lib/python3.8/site-packages/tensorflow/python/eager/def_function.py\u001b[0m in \u001b[0;36m__call__\u001b[0;34m(self, *args, **kwds)\u001b[0m\n\u001b[1;32m    887\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    888\u001b[0m       \u001b[0;32mwith\u001b[0m \u001b[0mOptionalXlaContext\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_jit_compile\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 889\u001b[0;31m         \u001b[0mresult\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_call\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwds\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    890\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    891\u001b[0m       \u001b[0mnew_tracing_count\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mexperimental_get_tracing_count\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/public/huangwei/miniconda3/envs/tensorflow/lib/python3.8/site-packages/tensorflow/python/eager/def_function.py\u001b[0m in \u001b[0;36m_call\u001b[0;34m(self, *args, **kwds)\u001b[0m\n\u001b[1;32m    915\u001b[0m       \u001b[0;31m# In this case we have created variables on the first call, so we run the\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    916\u001b[0m       \u001b[0;31m# defunned version which is guaranteed to never create variables.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 917\u001b[0;31m       \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_stateless_fn\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwds\u001b[0m\u001b[0;34m)\u001b[0m  \u001b[0;31m# pylint: disable=not-callable\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    918\u001b[0m     \u001b[0;32melif\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_stateful_fn\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    919\u001b[0m       \u001b[0;31m# Release the lock early so that multiple threads can perform the call\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/public/huangwei/miniconda3/envs/tensorflow/lib/python3.8/site-packages/tensorflow/python/eager/function.py\u001b[0m in \u001b[0;36m__call__\u001b[0;34m(self, *args, **kwargs)\u001b[0m\n\u001b[1;32m   3021\u001b[0m       (graph_function,\n\u001b[1;32m   3022\u001b[0m        filtered_flat_args) = self._maybe_define_function(args, kwargs)\n\u001b[0;32m-> 3023\u001b[0;31m     return graph_function._call_flat(\n\u001b[0m\u001b[1;32m   3024\u001b[0m         filtered_flat_args, captured_inputs=graph_function.captured_inputs)  # pylint: disable=protected-access\n\u001b[1;32m   3025\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/public/huangwei/miniconda3/envs/tensorflow/lib/python3.8/site-packages/tensorflow/python/eager/function.py\u001b[0m in \u001b[0;36m_call_flat\u001b[0;34m(self, args, captured_inputs, cancellation_manager)\u001b[0m\n\u001b[1;32m   1958\u001b[0m         and executing_eagerly):\n\u001b[1;32m   1959\u001b[0m       \u001b[0;31m# No tape is watching; skip to running the function.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1960\u001b[0;31m       return self._build_call_outputs(self._inference_function.call(\n\u001b[0m\u001b[1;32m   1961\u001b[0m           ctx, args, cancellation_manager=cancellation_manager))\n\u001b[1;32m   1962\u001b[0m     forward_backward = self._select_forward_and_backward_functions(\n",
      "\u001b[0;32m/public/huangwei/miniconda3/envs/tensorflow/lib/python3.8/site-packages/tensorflow/python/eager/function.py\u001b[0m in \u001b[0;36mcall\u001b[0;34m(self, ctx, args, cancellation_manager)\u001b[0m\n\u001b[1;32m    589\u001b[0m       \u001b[0;32mwith\u001b[0m \u001b[0m_InterpolateFunctionError\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    590\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0mcancellation_manager\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 591\u001b[0;31m           outputs = execute.execute(\n\u001b[0m\u001b[1;32m    592\u001b[0m               \u001b[0mstr\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msignature\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mname\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    593\u001b[0m               \u001b[0mnum_outputs\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_num_outputs\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/public/huangwei/miniconda3/envs/tensorflow/lib/python3.8/site-packages/tensorflow/python/eager/execute.py\u001b[0m in \u001b[0;36mquick_execute\u001b[0;34m(op_name, num_outputs, inputs, attrs, ctx, name)\u001b[0m\n\u001b[1;32m     57\u001b[0m   \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     58\u001b[0m     \u001b[0mctx\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mensure_initialized\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 59\u001b[0;31m     tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name,\n\u001b[0m\u001b[1;32m     60\u001b[0m                                         inputs, attrs, num_outputs)\n\u001b[1;32m     61\u001b[0m   \u001b[0;32mexcept\u001b[0m \u001b[0mcore\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_NotOkStatusException\u001b[0m \u001b[0;32mas\u001b[0m \u001b[0me\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mKeyboardInterrupt\u001b[0m: "
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "time: 19min 3s (started: 2021-08-04 20:15:12 +08:00)\n"
     ]
    }
   ],
   "source": [
    "vocab_size = 20000\n",
    "sequence_length = 600\n",
    "embed_dim = 256\n",
    "num_heads = 2\n",
    "dense_dim = 32\n",
    "\n",
    "inputs = keras.Input(shape=(None,), dtype=\"int64\")\n",
    "\n",
    "# 看这里！\n",
    "x = PositionalEmbedding(sequence_length, vocab_size, embed_dim)(inputs) \n",
    "x = TransformerEncoder(embed_dim, dense_dim, num_heads)(x)\n",
    "x = layers.GlobalMaxPooling1D()(x)\n",
    "x = layers.Dropout(0.5)(x)\n",
    "outputs = layers.Dense(1, activation=\"sigmoid\")(x)\n",
    "\n",
    "model = keras.Model(inputs, outputs)\n",
    "\n",
    "model.compile(optimizer=\"rmsprop\",\n",
    "              loss=\"binary_crossentropy\",\n",
    "              metrics=[\"accuracy\"])\n",
    "\n",
    "model.summary()\n",
    "\n",
    "callbacks = [keras.callbacks.ModelCheckpoint(\"full_transformer_encoder.keras\",\n",
    "                                             save_best_only=True)]\n",
    "\n",
    "model.fit(int_train_ds, validation_data=int_val_ds, epochs=20, callbacks=callbacks)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "municipal-exploration",
   "metadata": {},
   "outputs": [],
   "source": [
    "model = keras.models.load_model(\"full_transformer_encoder.keras\",\n",
    "                                custom_objects={\"TransformerEncoder\": TransformerEncoder,\n",
    "                                                \"PositionalEmbedding\": PositionalEmbedding})\n",
    "\n",
    "print(f\"Test acc: {model.evaluate(int_test_ds)[1]:.3f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "packed-current",
   "metadata": {},
   "source": [
    "我们达到了 88.3% 的测试准确率，这是一个坚实的改进，清楚地证明了词序信息对于文本分类的价值。 这是迄今为止我们最好的序列模型！ 然而，它仍然比词袋方法低一个档次。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "smooth-anthony",
   "metadata": {},
   "source": [
    "### 何时在词袋模型上使用序列模型？"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "scenic-mobile",
   "metadata": {},
   "source": [
    "您有时可能会听到词袋方法已经过时，并且无论您正在查看什么任务或数据集，基于 Transformer 的序列模型都是可行的方法。 这绝对不是这种情况：在许多情况下，在二元组袋顶部的一小堆层仍然是一种完全有效且相关的 Dense 方法。 事实上，在本章中我们在 IMDB 数据集上尝试的各种技术中，迄今为止表现最好的是 bag-of-bigrams！\n",
    "\n",
    "那么，什么时候您应该更喜欢一种方法而不是另一种方法？\n",
    "\n",
    "2017 年，我和我的团队对各种文本分类技术在许多不同类型的文本数据集上的性能进行了系统分析，我们发现了一个非凡且令人惊讶的经验法则来决定是否使用词袋模型 或序列模型。 各种黄金常数。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "smoking-oregon",
   "metadata": {},
   "source": [
    "事实证明，在处理新的文本分类任务时，您应该密切注意训练数据中的样本数与每个样本的平均单词数之间的比率。 如果该比率很小（小于 1,500），那么bag-of-bigrams 模型的性能会更好（而且作为奖励，它的训练和迭代速度也会快得多）。 如果该比率高于 1,500，那么您应该使用序列模型。 换句话说，当有大量训练数据可用且每个样本相对较短时，序列模型效果最好。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "speaking-audio",
   "metadata": {},
   "source": [
    "![](https://tva1.sinaimg.cn/large/008i3skNgy1gt50v7pcsoj31as0iodi0.jpg)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "thermal-penguin",
   "metadata": {},
   "source": [
    "因此，如果您要对 1,000 字长的文档进行分类，并且其中有 100,000 个，那么您应该使用二元模型（比率：100）。 如果您对平均 40 字长的推文进行分类，并且您有 50,000 条推文，那么您还应该使用二元模型（比率：1,250）。 但是，如果您将数据集大小增加到 500,000 条推文，则使用 Transformer 编码器（比率：12,500）。 IMDB 影评分类任务怎么样？ 我们有 20,000 个训练样本，平均字数为 233，因此我们的经验法则指向二元模型——这证实了我们在实践中的发现。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "thrown-monthly",
   "metadata": {},
   "source": [
    "因此，如果您要对 1,000 字长的文档进行分类，并且其中有 100,000 个，那么您应该使用二元模型（比率：100）。 如果您对平均 40 字长的推文进行分类，并且您有 50,000 条推文，那么您还应该使用二元模型（比率：1,250）。 但是，如果您将数据集大小增加到 500,000 条推文，则使用 Transformer 编码器（比率：12,500）。 IMDB 影评分类任务怎么样？ 我们有 20,000 个训练样本，平均字数为 233，因此我们的经验法则指向二元模型——这证实了我们在实践中的发现。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "designing-convergence",
   "metadata": {},
   "source": [
    "这在直觉上是有道理的：序列模型的输入代表一个更丰富、更复杂的空间，因此需要更多的数据来绘制该空间——同时，一组简单的术语是一个非常简单的空间，您可以训练逻辑回归 最重要的是仅使用数百或数千个样本。 此外，样本越短，模型丢弃它包含的任何信息的能力就越少——尤其是词序变得更加重要，丢弃它会产生歧义。 句子“这部电影是炸弹”和“这部电影是炸弹”具有非常接近的一元表示，这可能会混淆词袋模型，但序列模型可以分辨哪个是负面的，哪个是正面的。 使用更长的样本，单词统计将变得更加可靠，并且仅从单词直方图中可以更明显地看出主题或情绪。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "random-traffic",
   "metadata": {},
   "source": [
    "现在，请记住，这个启发式规则是专门为文本分类开发的，它可能不一定适用于其他 NLP 任务——例如，在机器翻译方面，与 RNN 相比，Transformer 特别适用于非常长的序列。 我们的启发式也只是一个经验法则，而不是科学定律，所以希望它在大多数时间都有效，但不一定每次都有效。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "informed-puzzle",
   "metadata": {},
   "source": [
    "## 超越文本分类：序列到序列学习"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "classified-crest",
   "metadata": {},
   "source": [
    "到目前为止，您已经拥有处理大多数自然语言处理任务所需的所有工具。 但是，到目前为止，您只有这些工具可以解决一个问题：文本分类。 这是一个非常流行的用例，但 NLP 的分类还有很多。 在本节中，您将通过学习序列到序列模型来加深您的专业知识。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "engaged-badge",
   "metadata": {},
   "source": [
    "序列到序列模型将序列作为输入（通常是句子或段落）并将其翻译成不同的序列。 这是许多 NLP 最成功应用的核心任务，例如（见图 11.12）：\n",
    "* 机器翻译：将源语言中的段落转换为目标语言中的对应段落。\n",
    "* 文本摘要：将长文档转换为保留最重要信息的较短版本。\n",
    "* 问答：将输入的问题转换为其答案。 聊天机器人：将对话提示转换为对此提示的回复，或将对话的历史记录转换为对话中的下一个回复。\n",
    "* 文本生成：将文本提示转换为完成提示的段落。\n",
    "* 等等。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fallen-brazil",
   "metadata": {},
   "source": [
    "图 11.12 描述了序列到序列模型背后的通用模板。 培训期间：\n",
    "* 编码器模型将源序列转换为中间表示。\n",
    "* 通过查看 i 个先前的标记（0 到 i - 1）和编码的源序列，解码器被训练来预测目标序列中的下一个标记。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "acquired-nightmare",
   "metadata": {},
   "source": [
    "![](https://tva1.sinaimg.cn/large/008i3skNgy1gt515nklwsj317j0u0n14.jpg)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "antique-checkout",
   "metadata": {},
   "source": [
    "在推理过程中，我们无法访问目标序列——我们试图从头开始预测它。 我们必须一次生成一个令牌：\n",
    "* 首先，我们从编码器获得编码的源序列。\n",
    "* 解码器首先查看编码的源序列以及初始“种子”标记（例如 string ），并使用它们来预测“[start]”序列中的第一个真实标记。\n",
    "* 到目前为止的预测序列被反馈到解码器，解码器生成下一个标记，依此类推，直到生成停止标记（例如字符串“[end]”）。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "recorded-parker",
   "metadata": {},
   "source": [
    "到目前为止，您学到的所有知识都可以重新用于构建这种新型模型。 让我们深入了解。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "exempt-challenge",
   "metadata": {},
   "source": [
    "### 机器翻译示例"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "exempt-monitor",
   "metadata": {},
   "source": [
    "我们将在机器翻译任务上演示序列到序列建模。 机器翻译正是 Transformer 的开发目的！ 我们将从循环序列模型开始，然后跟进完整的 Transformer 架构。\n",
    "\n",
    "我们将使用 Anki (www.manythings.org/anki/) 提供的英语到西班牙语翻译数据集。 让我们下载它："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "excellent-facing",
   "metadata": {},
   "outputs": [],
   "source": [
    "!wget http://storage.googleapis.com/download.tensorflow.org/data/spa-eng.zip\n",
    "!unzip -q spa-eng.zip"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "incorporated-leone",
   "metadata": {},
   "source": [
    "该文本文件每行包含一个示例：一个英语句子，后跟一个制表符，后跟相应的西班牙语句子。 让我们解析这个文件。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "finnish-sandwich",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "time: 525 ms (started: 2021-08-05 11:15:31 +08:00)\n"
     ]
    }
   ],
   "source": [
    "text_file = \"data/spa-eng/spa.txt\"\n",
    "with open(text_file) as f:\n",
    "    lines = f.read().split(\"\\n\")[:-1]\n",
    "    \n",
    "text_pairs = []\n",
    "\n",
    "# 迭代文件中的行。\n",
    "for line in lines:\n",
    "#     每行包含一个英语短语及其西班牙语翻译，以制表符分隔。\n",
    "    english, spanish = line.split(\"\\t\")\n",
    "#     我们在西班牙语句子前面加上“[start]”和“[end]”，以匹配图 11.12 中的模板。\n",
    "    spanish = \"[start] \" + spanish + \" [end]\" \n",
    "    text_pairs.append((english, spanish))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "appropriate-substance",
   "metadata": {},
   "source": [
    "我们的 text_pairs 看起来像这样："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "verified-burns",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "('You might want to have someone look into that matter.', '[start] Puede que quieras que alguien investigue ese asunto. [end]')\n",
      "time: 1.94 ms (started: 2021-08-05 11:15:33 +08:00)\n"
     ]
    }
   ],
   "source": [
    "import random\n",
    "print(random.choice(text_pairs))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "coordinated-airfare",
   "metadata": {},
   "source": [
    "让我们将它们打乱，然后分成通常的训练、验证和测试集："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "injured-report",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "time: 195 ms (started: 2021-08-05 11:15:35 +08:00)\n"
     ]
    }
   ],
   "source": [
    "import random\n",
    "\n",
    "random.shuffle(text_pairs)\n",
    "\n",
    "num_val_samples = int(0.15 * len(text_pairs))\n",
    "num_train_samples = len(text_pairs) - 2 * num_val_samples\n",
    "\n",
    "train_pairs = text_pairs[:num_train_samples]\n",
    "val_pairs = text_pairs[num_train_samples:num_train_samples + num_val_samples]\n",
    "test_pairs = text_pairs[num_train_samples + num_val_samples:]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "warming-myanmar",
   "metadata": {},
   "source": [
    "接下来，让我们准备两个单独的 TextVectorization 层，一个用于英语，一个用于西班牙语。 我们将需要自定义字符串的预处理方式：\n",
    "* 我们需要保留我们插入的“[start]”和“[end]”标记。 默认情况下，字符 [and] 将被删除，但我们希望保留它们以便我们可以区分单词“start”和开始标记“[start]”。\n",
    "* 标点符号因语言而异！ 在西班牙语 TextVectorization 层中，如果我们要去除标点字符，我们还需要去除字符。\n",
    "\n",
    "请注意，对于非玩具翻译模型，我们会将标点字符视为单独的标记而不是剥离它们，因为我们希望能够生成标点正确的句子。 在我们的例子中，为简单起见，我们将去掉所有标点符号。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "correct-acquisition",
   "metadata": {},
   "source": [
    "> 清单 11.28 向量化英语和西班牙语文本对"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "civil-visit",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "time: 1min 22s (started: 2021-08-05 11:15:43 +08:00)\n"
     ]
    }
   ],
   "source": [
    "from keras.layers.experimental.preprocessing import TextVectorization\n",
    "import tensorflow as tf\n",
    "import string\n",
    "import re\n",
    "\n",
    "# 为西班牙语 TextVectorization 层准备一个自定义字符串标准化函数：\n",
    "# 它保留并删除（以及来自 strings.punctuation 的所有其他 [ ] ¿ 字符）。\n",
    "strip_chars = string.punctuation + \"¿\"\n",
    "strip_chars = strip_chars.replace(\"[\", \"\")\n",
    "strip_chars = strip_chars.replace(\"]\", \"\")\n",
    "def custom_standardization(input_string):\n",
    "    lowercase = tf.strings.lower(input_string)\n",
    "    return tf.strings.regex_replace(lowercase, f\"[{re.escape(strip_chars)}]\", \"\")\n",
    "\n",
    "# 为简单起见，我们将只查看每种语言的前 15,000 个单词，并将句子限制为 20 个单词。\n",
    "vocab_size = 15000\n",
    "sequence_length = 20\n",
    "\n",
    "# 英语层\n",
    "source_vectorization = TextVectorization( \n",
    "    max_tokens=vocab_size,\n",
    "    output_mode=\"int\",\n",
    "    output_sequence_length=sequence_length,\n",
    "    )\n",
    "\n",
    "# 西班牙语层\n",
    "target_vectorization = TextVectorization( \n",
    "    max_tokens=vocab_size,\n",
    "    output_mode=\"int\",\n",
    "#     生成具有一个额外标记的西班牙语句子，因为我们需要在训练期间将句子偏移一级。\n",
    "    output_sequence_length=sequence_length + 1, \n",
    "    standardize=custom_standardization,\n",
    ")\n",
    "\n",
    "train_english_texts = [pair[0] for pair in train_pairs]\n",
    "train_spanish_texts = [pair[1] for pair in train_pairs]\n",
    "\n",
    "# 学习每种语言的词汇。\n",
    "source_vectorization.adapt(train_english_texts) \n",
    "target_vectorization.adapt(train_spanish_texts)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "sunrise-allergy",
   "metadata": {},
   "source": [
    "最后，我们可以将我们的数据变成一个管道。 我们希望它返回一个元组 tf.data (inputs, target) input 其中是一个带有两个键的字典，“encoder_inputs”（英语句子）和“decoder_inputs”（西班牙语句子），并且是偏移一的西班牙语句子 前进目标。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "careful-planet",
   "metadata": {},
   "source": [
    "> 清单 11.29 为翻译任务准备训练和验证数据集"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "hourly-colon",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "time: 2.91 s (started: 2021-08-05 11:19:37 +08:00)\n"
     ]
    }
   ],
   "source": [
    "batch_size = 64\n",
    "\n",
    "def format_dataset(eng, spa):\n",
    "    eng = source_vectorization(eng)\n",
    "    spa = target_vectorization(spa)\n",
    "    return ({\"english\": eng,\n",
    "             \"spanish\": spa[:, :-1], #输入的西班牙语句子不包括最后一个标记，以保持输入和目标的长度相同。\n",
    "            }, spa[:, 1:]) # 目标西班牙语句子领先一步。 两者的长度仍然相同（20 个字）。\n",
    "\n",
    "def make_dataset(pairs):\n",
    "    eng_texts, spa_texts = zip(*pairs)\n",
    "    eng_texts = list(eng_texts)\n",
    "    spa_texts = list(spa_texts)\n",
    "    dataset = tf.data.Dataset.from_tensor_slices((eng_texts, spa_texts))\n",
    "    dataset = dataset.batch(batch_size)\n",
    "    dataset = dataset.map(format_dataset)\n",
    "    return dataset.shuffle(2048).prefetch(16).cache() #使用内存缓存来加速预处理。\n",
    "\n",
    "train_ds = make_dataset(train_pairs)\n",
    "val_ds = make_dataset(val_pairs)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "popular-taylor",
   "metadata": {},
   "source": [
    "这是我们的数据集输出的样子："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "accessory-covering",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "inputs['english'].shape: (64, 20)\n",
      "inputs['spanish'].shape: (64, 20)\n",
      "targets.shape: (64, 20)\n",
      "time: 1.64 s (started: 2021-08-05 11:20:39 +08:00)\n"
     ]
    }
   ],
   "source": [
    "for inputs, targets in train_ds.take(1):\n",
    "    print(f\"inputs['english'].shape: {inputs['english'].shape}\")\n",
    "    print(f\"inputs['spanish'].shape: {inputs['spanish'].shape}\")\n",
    "    print(f\"targets.shape: {targets.shape}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "indonesian-tablet",
   "metadata": {},
   "source": [
    "数据现已准备就绪，是时候构建一些模型了。 在转到 Transformer 之前，我们将从循环序列到序列模型开始。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "biblical-chocolate",
   "metadata": {},
   "source": [
    "### 使用 RNN 进行序列到序列学习"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "respective-expense",
   "metadata": {},
   "source": [
    "从 2015 年到 2017 年，循环神经网络主导了序列到序列学习，之后被 Transformer 超越。 它们是许多现实世界机器翻译系统的基础——如第 10 章所述，大约 2017 年的谷歌翻译由七个大型 LSTM 层的堆栈提供支持。 今天仍然值得学习这种方法，因为它提供了一个简单的切入点来理解序列到序列模型。\n",
    "\n",
    "使用 RNN 将序列转换为另一个序列的最简单、天真的方法是在每个时间步保持 RNN 的输出——在 Keras 中，它看起来像这样："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "meaning-chart",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "time: 1.16 s (started: 2021-08-05 11:24:19 +08:00)\n"
     ]
    }
   ],
   "source": [
    "from tensorflow import keras\n",
    "from tensorflow.keras import layers\n",
    "\n",
    "inputs = layers.Input(shape=(sequence_length,), dtype=\"int64\")\n",
    "x = layers.Embedding(input_dim=vocab_size, output_dim=128)(inputs)\n",
    "x = layers.LSTM(32, return_sequences=True)(x)\n",
    "outputs = layers.Dense(vocab_size, activation=\"softmax\")(x)\n",
    "model = keras.Model(inputs, outputs)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "preliminary-empty",
   "metadata": {},
   "source": [
    "但是，这种方法有两个主要问题：\n",
    "* 目标序列必须始终与源序列长度相同。 在实践中，这种情况很少发生。 从技术上讲，这并不重要，因为您始终可以填充源序列或目标序列以使其长度匹配。\n",
    "* 由于 RNN 的循序渐进特性，该模型将只查看源序列中的标记 0…N，以便预测目标序列中的标记 N。 这种限制使得这种设置不适用于大多数任务，尤其是翻译。 考虑将“今天天气很好”翻译成法语——那就是“Il fait beau aujourd’hui”。 您需要能够仅从“The”预测“Il”，仅从“The weather”等预测“Il fait”，这是不可能的。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "immediate-giving",
   "metadata": {},
   "source": [
    "如果你是一名人工翻译，你会先阅读整个源句子，然后再开始翻译。 如果您正在处理具有截然不同的词序的语言，例如英语和日语，这一点尤其重要。 而这正是标准序列到序列模型所做的。\n",
    "\n",
    "在正确的序列到序列设置中（参见图 11.13），您将首先使用 RNN（编码器）将整个源序列转换为单个向量（或一组向量）。 这可能是 RNN 的最后一个输出，或者是其最终的内部状态向量。 然后你可以使用这个向量（或多个向量）作为另一个 RNN（解码器）的 ，它会查看目标序列中的初始状态元素 0… N，并尝试预测目标序列中的第 N+1 步。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fitted-handbook",
   "metadata": {},
   "source": [
    "![](https://tva1.sinaimg.cn/large/008i3skNgy1gt5r4g63ksj31c20m6tbq.jpg)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "streaming-candy",
   "metadata": {},
   "source": [
    "让我们使用基于 GRU 的编码器和解码器在 Keras 中实现这一点。 选择 GRU 而不是 LSTM 让事情变得更简单，因为 GRU 只有一个状态向量，而 LSTM 有多个。 让我们从编码器开始："
   ]
  },
  {
   "cell_type": "markdown",
   "id": "varied-prayer",
   "metadata": {},
   "source": [
    "> 清单 11.30 基于 GRU 的编码器"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "amino-operator",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "time: 6.33 s (started: 2021-08-05 11:31:45 +08:00)\n"
     ]
    }
   ],
   "source": [
    "from tensorflow import keras\n",
    "from tensorflow.keras import layers\n",
    "\n",
    "embed_dim = 256\n",
    "latent_dim = 1024\n",
    "\n",
    "# 英文源句放在这里。 指定输入的名称使我们能够使用具有输入字典的 fit() 模型。\n",
    "source = keras.Input(shape=(None,), dtype=\"int64\", name=\"english\") \n",
    "# 不要忘记屏蔽：这在此设置中至关重要。\n",
    "x = layers.Embedding(vocab_size, embed_dim, mask_zero=True)(source) \n",
    "# 我们编码的源语句是双向 GRU 的最后一个输出。\n",
    "encoded_source = layers.Bidirectional(layers.GRU(latent_dim), merge_mode=\"sum\")(x)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "homeless-hunger",
   "metadata": {},
   "source": [
    "接下来，让我们添加解码器——一个简单的 GRU 层，它将编码的源语句作为初始状态。 最重要的是，我们添加了一个 Dense 层，为每个输出步骤生成西班牙语词汇表的概率分布。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "manufactured-accident",
   "metadata": {},
   "source": [
    "> 清单 11.31 基于 GRU 的解码器和端到端模型"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "white-substitute",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "time: 3.43 s (started: 2021-08-05 11:37:50 +08:00)\n"
     ]
    }
   ],
   "source": [
    "# 西班牙语目标句放在这里。\n",
    "past_target = keras.Input(shape=(None,), dtype=\"int64\", name=\"spanish\") \n",
    "# 不要忘记掩码。\n",
    "x = layers.Embedding(vocab_size, embed_dim, mask_zero=True)(past_target) \n",
    "decoder_gru = layers.GRU(latent_dim, return_sequences=True)\n",
    "# 编码的源语句作为解码器 GRU 的初始状态。\n",
    "x = decoder_gru(x, initial_state=encoded_source) \n",
    "x = layers.Dropout(0.5)(x)\n",
    "# 预测下一个令牌。\n",
    "target_next_step = layers.Dense(vocab_size, activation=\"softmax\")(x) \n",
    "# 端到端模型：将源句和目标句映射到未来一步的目标句。\n",
    "seq2seq_rnn = keras.Model([source, past_target], target_next_step)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "adult-reaction",
   "metadata": {},
   "source": [
    "在训练期间，解码器将整个目标序列作为输入，但由于 RNN 的逐步性质，它只查看输入中的标记 0… N 来预测输出中的标记 N（对应于下一个 序列中的标记，因为输出旨在偏移一个步骤）。 这意味着我们只使用过去的信息来预测未来——我们应该这样做，否则，我们会作弊，我们的模型在推理时将无法工作。\n",
    "\n",
    "让我们开始训练："
   ]
  },
  {
   "cell_type": "markdown",
   "id": "identical-orange",
   "metadata": {},
   "source": [
    "> 清单 11.32 训练我们的循环序列到序列模型"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "framed-brother",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/15\n",
      "1302/1302 [==============================] - 222s 141ms/step - loss: 1.6373 - accuracy: 0.4182 - val_loss: 1.3037 - val_accuracy: 0.5109\n",
      "Epoch 2/15\n",
      "1302/1302 [==============================] - 176s 135ms/step - loss: 1.3139 - accuracy: 0.5294 - val_loss: 1.1425 - val_accuracy: 0.5719\n",
      "Epoch 3/15\n",
      "1302/1302 [==============================] - 174s 133ms/step - loss: 1.1727 - accuracy: 0.5779 - val_loss: 1.0652 - val_accuracy: 0.6032\n",
      "Epoch 4/15\n",
      "1302/1302 [==============================] - 173s 133ms/step - loss: 1.0830 - accuracy: 0.6096 - val_loss: 1.0293 - val_accuracy: 0.6204\n",
      "Epoch 5/15\n",
      "1302/1302 [==============================] - 173s 133ms/step - loss: 1.0349 - accuracy: 0.6340 - val_loss: 1.0152 - val_accuracy: 0.6309\n",
      "Epoch 6/15\n",
      "1302/1302 [==============================] - 173s 133ms/step - loss: 1.0050 - accuracy: 0.6518 - val_loss: 1.0136 - val_accuracy: 0.6363\n",
      "Epoch 7/15\n",
      "1302/1302 [==============================] - 173s 133ms/step - loss: 0.9854 - accuracy: 0.6650 - val_loss: 1.0128 - val_accuracy: 0.6389\n",
      "Epoch 8/15\n",
      "1302/1302 [==============================] - 175s 135ms/step - loss: 0.9728 - accuracy: 0.6750 - val_loss: 1.0153 - val_accuracy: 0.6399\n",
      "Epoch 9/15\n",
      "1302/1302 [==============================] - 175s 134ms/step - loss: 0.9625 - accuracy: 0.6829 - val_loss: 1.0195 - val_accuracy: 0.6420\n",
      "Epoch 10/15\n",
      "1302/1302 [==============================] - 175s 134ms/step - loss: 0.9550 - accuracy: 0.6888 - val_loss: 1.0218 - val_accuracy: 0.6437\n",
      "Epoch 11/15\n",
      "1302/1302 [==============================] - 176s 135ms/step - loss: 0.9494 - accuracy: 0.6931 - val_loss: 1.0236 - val_accuracy: 0.6436\n",
      "Epoch 12/15\n",
      "1302/1302 [==============================] - 176s 135ms/step - loss: 0.9464 - accuracy: 0.6956 - val_loss: 1.0292 - val_accuracy: 0.6445\n",
      "Epoch 13/15\n",
      "1302/1302 [==============================] - 175s 134ms/step - loss: 0.9435 - accuracy: 0.6980 - val_loss: 1.0302 - val_accuracy: 0.6446\n",
      "Epoch 14/15\n",
      "1302/1302 [==============================] - 173s 133ms/step - loss: 0.9428 - accuracy: 0.6995 - val_loss: 1.0314 - val_accuracy: 0.6446\n",
      "Epoch 15/15\n",
      "1302/1302 [==============================] - 175s 134ms/step - loss: 0.9425 - accuracy: 0.6998 - val_loss: 1.0313 - val_accuracy: 0.6447\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<tensorflow.python.keras.callbacks.History at 0x7fb8b0cef700>"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "time: 44min 24s (started: 2021-08-05 11:39:36 +08:00)\n"
     ]
    }
   ],
   "source": [
    "seq2seq_rnn.compile(optimizer=\"rmsprop\",\n",
    "                    loss=\"sparse_categorical_crossentropy\",\n",
    "                    metrics=[\"accuracy\"])\n",
    "seq2seq_rnn.fit(train_ds, epochs=15, validation_data=val_ds)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "necessary-prerequisite",
   "metadata": {},
   "source": [
    "我们选择准确性作为在训练期间监控验证集性能的粗略方法。 我们达到了 64% 的准确率：平均而言，该模型在 64% 的时间内正确预测了西班牙语句子中的下一个单词。 然而，在实践中，下一个标记的准确性并不是机器翻译模型的重要指标，特别是因为它假设在预测标记 N+1 时已经知道从 0 到 N 的正确目标标记——实际上，在 推断，您是从头开始生成目标句子，因此您不能依赖先前生成的标记是 100% 正确的。 如果您继续在现实世界的机器翻译系统上工作，您可能会使用 BLEU 分数来评估您的模型——一种查看整个生成序列的指标，并且似乎与人类对翻译质量的感知密切相关。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "affected-apparel",
   "metadata": {},
   "source": [
    "最后，让我们使用我们的模型进行推理——我们将在测试集中挑选一些句子并检查我们的模型如何翻译它们。 我们从种子标记“[start]”开始，并将其与编码的英文源句子一起输入解码器模型。 我们检索下一个令牌预测。 我们将其重新注入解码器，重复，在每次迭代中采样一个新的目标标记，直到我们到达“[end]”或达到最大句子长度。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "urban-extension",
   "metadata": {},
   "source": [
    "> 清单 11.33 使用我们的 RNN 编码器和解码器翻译新句子"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "intense-season",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "-\n",
      "He is still in bed.\n",
      "[start] Él está en cama [end]\n",
      "-\n",
      "The hotel has a pleasant atmosphere.\n",
      "[start] el hotel tiene una [UNK] [end]\n",
      "-\n",
      "I don't believe such things exist.\n",
      "[start] no creo que las cosas de cosas [end]\n",
      "-\n",
      "What time should I go to the airport?\n",
      "[start] a qué hora debería ir al aeropuerto [end]\n",
      "-\n",
      "His new book will appear next month.\n",
      "[start] su nuevo libro el día que viene [end]\n",
      "-\n",
      "We seem to be trapped.\n",
      "[start] parece estar en lo que me [UNK] [end]\n",
      "-\n",
      "I can do it in half the time.\n",
      "[start] puedo hacerlo en la hora [end]\n",
      "-\n",
      "The boss is an open person.\n",
      "[start] el hombre es una persona muy difícil [end]\n",
      "-\n",
      "Send him in.\n",
      "[start] [UNK] en él [end]\n",
      "-\n",
      "I don't speak Swedish.\n",
      "[start] no hablo nada [end]\n",
      "-\n",
      "Can he speak Japanese?\n",
      "[start] Él sabe hablar japonés [end]\n",
      "-\n",
      "Do you think I should go by myself?\n",
      "[start] crees que debería ir solo [end]\n",
      "-\n",
      "I still love this bicycle.\n",
      "[start] todavía me encanta esta bicicleta [end]\n",
      "-\n",
      "Picasso painted this picture in 1950.\n",
      "[start] [UNK] este [UNK] de la en la mano [end]\n",
      "-\n",
      "I'm not crazy.\n",
      "[start] no estoy loco [end]\n",
      "-\n",
      "Tom's bored.\n",
      "[start] tom está de [UNK] [end]\n",
      "-\n",
      "How many people are there in this office?\n",
      "[start] cuántas personas hay en esta oficina [end]\n",
      "-\n",
      "To tell the truth, I have no money with me.\n",
      "[start] a decir verdad no me tengo dinero [end]\n",
      "-\n",
      "He came to ask us for our help.\n",
      "[start] Él vino a nosotros [UNK] para nosotros [end]\n",
      "-\n",
      "I can afford it.\n",
      "[start] puedo [UNK] [end]\n",
      "time: 30.9 s (started: 2021-08-05 12:30:58 +08:00)\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "\n",
    "# 准备一个 dict 将令牌索引预测转换为字符串令牌\n",
    "spa_vocab = target_vectorization.get_vocabulary()\n",
    "spa_index_lookup = dict(zip(range(len(spa_vocab)), spa_vocab)) \n",
    "\n",
    "max_decoded_sentence_length = 20\n",
    "\n",
    "def decode_sequence(input_sentence):\n",
    "    tokenized_input_sentence = source_vectorization([input_sentence])\n",
    "    decoded_sentence = \"[start]\" #种子令牌\n",
    "    for i in range(max_decoded_sentence_length):\n",
    "        tokenized_target_sentence = target_vectorization([decoded_sentence])\n",
    "    #     采样下一个令牌\n",
    "        next_token_predictions = seq2seq_rnn.predict([tokenized_input_sentence, \n",
    "                                                      tokenized_target_sentence])\n",
    "        sampled_token_index = np.argmax(next_token_predictions[0, i, :]) \n",
    "        sampled_token = spa_index_lookup[sampled_token_index]\n",
    "    #     将下一个标记预测转换为字符串并将其附加到生成的句子中\n",
    "        decoded_sentence += \" \" + sampled_token\n",
    "    #     退出条件：达到最大长度或采样一个停止字符。\n",
    "        if sampled_token == \"[end]\":\n",
    "            break\n",
    "    return decoded_sentence\n",
    "\n",
    "test_eng_texts = [pair[0] for pair in test_pairs]\n",
    "for _ in range(20):\n",
    "    input_sentence = random.choice(test_eng_texts)\n",
    "    print(\"-\")\n",
    "    print(input_sentence)\n",
    "    print(decode_sequence(input_sentence))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dynamic-philosophy",
   "metadata": {},
   "source": [
    "请注意，这种推理设置虽然非常简单，但效率相当低，因为我们每次对新单词进行采样时都会重新处理整个源句子和整个生成的目标句子。 在实际应用中，您将编码器和解码器分解为两个独立的模型，并且您的解码器在每次令牌采样迭代时只运行一个步骤，重用其先前的内部状态。\n",
    "\n",
    "这是我们的翻译结果。 对于玩具模型，它工作得很好，尽管它仍然会犯许多基本错误，如清单 11.34 所示。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "atlantic-gossip",
   "metadata": {},
   "source": [
    "> 清单 11.34 循环翻译模型的一些示例结果"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "accepted-tribute",
   "metadata": {},
   "outputs": [],
   "source": [
    "Who is in this room?\n",
    "[start] quién está en esta habitación [end]\n",
    "-\n",
    "That doesn't sound too dangerous.\n",
    "[start] eso no es muy difícil [end]\n",
    "-\n",
    "No one will stop me.\n",
    "[start] nadie me va a hacer [end]\n",
    "-\n",
    "Tom is friendly.\n",
    "[start] tom es un buen [UNK] [end]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "worthy-proposition",
   "metadata": {},
   "source": [
    "有很多方法可以改进这个玩具模型：我们可以为编码器和解码器使用一个深层的循环层（请注意，对于解码器，这会使状态管理更加复杂）。我们可以使用 LSTM 代替 GRU。等等。然而，除了这些调整之外，用于序列到序列学习的 RNN 方法还有一些基本的局限性：\n",
    "* 源序列表示必须完全保存在编码器状态向量中，这对您可以翻译的句子的大小和复杂性产生了重大限制。这有点像人类完全从记忆中翻译一个句子，在生成翻译时没有看两次源句子。\n",
    "* RNN 无法处理很长的序列，因为它们往往会逐渐忘记过去——当你到达任一序列中的第 100 个标记时，关于序列开始的信息就很少了。这意味着基于 RNN 的模型无法保持长期上下文，这对于翻译长文档至关重要。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "popular-crossing",
   "metadata": {},
   "source": [
    "这些限制导致机器学习社区采用 Transformer 架构来解决序列到序列问题。 让我们来看看。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "representative-agenda",
   "metadata": {},
   "source": [
    "### 使用 Transformer 进行序列到序列学习"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "interesting-mississippi",
   "metadata": {},
   "source": [
    "序列到序列学习是 Transformer 真正闪耀的任务。神经注意力使 Transformer 模型能够成功处理比 RNN 可以处理的更长、更复杂的序列。\n",
    "\n",
    "作为一名将英语翻译成西班牙语的人，你不会一次一个单词读一个英语句子，记住它的意思，然后一次一个单词生成西班牙语句子。这可能适用于五个字的句子，但不太可能适用于整个段落。相反，您可能希望在源句子和正在进行的翻译之间来回切换，并在写下翻译的不同部分时“注意”源中的不同单词。\n",
    "\n",
    "这正是您可以通过神经注意力和 Transformers 实现的目标。您已经熟悉 Transformer 编码器，它使用自注意力来生成输入序列中每个标记的上下文感知表示。在序列到序列的 Transformer 中，Transformer 编码器自然会扮演编码器的角色，它读取源序列并生成它的编码表示。然而，与我们之前的 RNN 编码器不同，Transformer 编码器将编码表示保持在序列格式中：它是一个上下文感知嵌入向量的序列。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "charming-valuable",
   "metadata": {},
   "source": [
    "模型的后半部分是 Transformer 解码器。 就像 RNN 解码器一样，它读取目标序列中的标记 0… N 并尝试预测标记 N+1。 至关重要的是，在执行此操作时，它使用神经注意力来识别编码源语句中的哪些标记与它当前试图预测的目标标记最密切相关——这可能与人工翻译会做的没什么不同。 回想一下查询-键-值模型：在 Transformer 解码器中，目标序列作为一个注意力“查询”，用于更密切地关注源序列的不同部分（源序列同时扮演着键和 值）。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cutting-strike",
   "metadata": {},
   "source": [
    "**变压器解码器**\n",
    "\n",
    "图 11.14 显示了完整的序列到序列转换器。 查看解码器内部结构：您会发现它看起来与 Transformer 编码器非常相似，只是在应用于目标序列的自注意块和出口块的密集层之间插入了一个额外的注意块。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "parental-supervisor",
   "metadata": {},
   "source": [
    "![](https://tva1.sinaimg.cn/large/008i3skNgy1gt5siz8oh3j30vi0t8din.jpg)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "residential-lexington",
   "metadata": {},
   "source": [
    "让我们实施它。 与 TransformerEncoder 一样，我们将使用 Layer 子类。 在我们关注 call() 方法（动作发生的地方）之前，让我们首先定义类构造函数，其中包含我们将需要的层。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "strategic-polyester",
   "metadata": {},
   "source": [
    "> 清单 11.35 TransformerDecoder"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "noble-green",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "time: 1.65 ms (started: 2021-08-05 12:26:55 +08:00)\n"
     ]
    }
   ],
   "source": [
    "class TransformerDecoder(layers.Layer):\n",
    "    def __init__(self, embed_dim, dense_dim, num_heads, **kwargs):\n",
    "        super().__init__(**kwargs)\n",
    "        self.embed_dim = embed_dim\n",
    "        self.dense_dim = dense_dim\n",
    "        self.num_heads = num_heads\n",
    "        self.attention_1 = layers.MultiHeadAttention(num_heads=num_heads, key_dim=embed_dim)\n",
    "        self.attention_2 = layers.MultiHeadAttention(num_heads=num_heads, key_dim=embed_dim)\n",
    "        self.dense_proj = keras.Sequential([layers.Dense(dense_dim, activation=\"relu\"),\n",
    "                                            layers.Dense(embed_dim),])\n",
    "        self.layernorm_1 = layers.LayerNormalization()\n",
    "        self.layernorm_2 = layers.LayerNormalization()\n",
    "        self.layernorm_3 = layers.LayerNormalization()\n",
    "        \n",
    "#         此属性确保该层将其输入掩码传播到其输出：Keras 中的掩码是明确选择的。 \n",
    "#         如果您将掩码传递给未实现 compute_mask() 且未公开此 supports_masking 属性的图层，则会出现错误。\n",
    "        self.supports_masking = True \n",
    "    \n",
    "    def get_config(self):\n",
    "        config = super().get_config()\n",
    "        config.update({\"embed_dim\": self.embed_dim,\n",
    "                       \"num_heads\": self.num_heads,\n",
    "                       \"dense_dim\": self.dense_dim,})\n",
    "        return config"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "recent-paste",
   "metadata": {},
   "source": [
    "call() 方法几乎是图 11.14 中连接图的直接呈现。 但是还有一个额外的细节我们需要考虑：因果填充。 因果填充对于成功训练序列到序列转换器绝对至关重要。 与 RNN 不同，它一次查看其输入一个步骤，因此只能访问步骤 0..N 以生成输出步骤 N（即目标序列中的标记 N+1），TransformerDecoder 是 order- 不可知：它一次查看整个目标序列。 如果允许它使用其整个输入，它会简单地学习将输入步骤 N+1 复制到输出中的位置 N。 因此，该模型将达到完美的训练精度，但当然，在运行推理时，它完全没有用，因为超过 N 的输入步骤不可用。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "small-subscriber",
   "metadata": {},
   "source": [
    "解决方法很简单：我们将屏蔽成对注意力矩阵的上半部分，以防止模型关注来自未来的信息——生成目标时只应使用来自目标序列中标记 0..N 的信息 令牌 N+1（参见图 TODO）。 为此，我们将向 TransformerEncoder 添加一个 get_causal_attention_mask(self,inputs) 方法以检索我们可以传递给 MultiHeadAttention 层的注意力掩码。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "obvious-representative",
   "metadata": {},
   "source": [
    "> 清单 11.36 生成“因果掩码”的 TransformerDecoder 方法"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "considerable-cornwall",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "time: 1.4 ms (started: 2021-08-05 12:26:50 +08:00)\n"
     ]
    }
   ],
   "source": [
    "def get_causal_attention_mask(self, inputs):\n",
    "    input_shape = tf.shape(inputs)\n",
    "    batch_size, sequence_length = input_shape[0], input_shape[1]\n",
    "    i = tf.range(sequence_length)[:, tf.newaxis]\n",
    "    j = tf.range(sequence_length)\n",
    "    \n",
    "#     生成形状矩阵（sequence_length，sequence_length），一半为 1，另一半为 0\n",
    "    mask = tf.cast(i >= j, dtype=\"int32\")\n",
    "    \n",
    "#     沿批处理轴复制它以获得形状矩阵 (batch_size,序列长度，序列长度）\n",
    "    mask = tf.reshape(mask, (1, input_shape[1], input_shape[1])) \n",
    "    mult = tf.concat([tf.expand_dims(batch_size, -1),\n",
    "                      tf.constant([1, 1], dtype=tf.int32)], axis=0)\n",
    "    return tf.tile(mask, mult)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "owned-tolerance",
   "metadata": {},
   "source": [
    "现在，我们可以写下实现解码器前向传递的完整方法："
   ]
  },
  {
   "cell_type": "markdown",
   "id": "hidden-priority",
   "metadata": {},
   "source": [
    "> 清单 11.37 Transformer Decoder 的前向传递"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "square-coating",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "time: 1.48 ms (started: 2021-08-05 12:37:11 +08:00)\n"
     ]
    }
   ],
   "source": [
    "def call(self, inputs, encoder_outputs, mask=None):\n",
    "#     检索因果掩码\n",
    "    causal_mask = self.get_causal_attention_mask(inputs) \n",
    "    # 准备输入掩码（描述目标序列中的填充位置）\n",
    "    if mask is not None:\n",
    "        padding_mask = tf.cast(mask[:, tf.newaxis, :], dtype=\"int32\")\n",
    "        # 将两个蒙版合并在一起\n",
    "        padding_mask = tf.minimum(padding_mask, causal_mask) \n",
    "    attention_output_1 = self.attention_1(query=inputs, \n",
    "                                          value=inputs,\n",
    "                                          key=inputs,\n",
    "                                          # 将因果掩码传递给第一个注意力层，它对目标序列执行自注意力\n",
    "                                          attention_mask=causal_mask)\n",
    "    attention_output_1 = self.layernorm_1(inputs + attention_output_1)\n",
    "    attention_output_2 = self.attention_2(query=attention_output_1,\n",
    "                                          value=encoder_outputs,\n",
    "                                          key=encoder_outputs,\n",
    "                                          # 将组合掩码传递给第二个注意力层，它将源序列与目标序列相关联\n",
    "                                          attention_mask=padding_mask,)\n",
    "    attention_output_2 = self.layernorm_2(attention_output_1 + attention_output_2)\n",
    "    proj_output = self.dense_proj(attention_output_2)\n",
    "    \n",
    "    return self.layernorm_3(attention_output_2 + proj_output)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "id": "lyric-jersey",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "time: 4.13 ms (started: 2021-08-05 12:51:35 +08:00)\n"
     ]
    }
   ],
   "source": [
    "class TransformerDecoder(layers.Layer):\n",
    "    def __init__(self, embed_dim, dense_dim, num_heads, **kwargs):\n",
    "        super().__init__(**kwargs)\n",
    "        self.embed_dim = embed_dim\n",
    "        self.dense_dim = dense_dim\n",
    "        self.num_heads = num_heads\n",
    "        self.attention_1 = layers.MultiHeadAttention(num_heads=num_heads, key_dim=embed_dim)\n",
    "        self.attention_2 = layers.MultiHeadAttention(num_heads=num_heads, key_dim=embed_dim)\n",
    "        self.dense_proj = keras.Sequential([layers.Dense(dense_dim, activation=\"relu\"),\n",
    "                                            layers.Dense(embed_dim),])\n",
    "        self.layernorm_1 = layers.LayerNormalization()\n",
    "        self.layernorm_2 = layers.LayerNormalization()\n",
    "        self.layernorm_3 = layers.LayerNormalization()\n",
    "        \n",
    "#         此属性确保该层将其输入掩码传播到其输出：Keras 中的掩码是明确选择的。 \n",
    "#         如果您将掩码传递给未实现 compute_mask() 且未公开此 supports_masking 属性的图层，则会出现错误。\n",
    "        self.supports_masking = True \n",
    "    \n",
    "    def call(self, inputs, encoder_outputs, mask=None):\n",
    "    #     检索因果掩码\n",
    "        causal_mask = self.get_causal_attention_mask(inputs) \n",
    "        # 准备输入掩码（描述目标序列中的填充位置）\n",
    "        if mask is not None:\n",
    "            padding_mask = tf.cast(mask[:, tf.newaxis, :], dtype=\"int32\")\n",
    "            # 将两个蒙版合并在一起\n",
    "            padding_mask = tf.minimum(padding_mask, causal_mask) \n",
    "        attention_output_1 = self.attention_1(query=inputs, \n",
    "                                              value=inputs,\n",
    "                                              key=inputs,\n",
    "                                              # 将因果掩码传递给第一个注意力层，它对目标序列执行自注意力\n",
    "                                              attention_mask=causal_mask)\n",
    "        attention_output_1 = self.layernorm_1(inputs + attention_output_1)\n",
    "        attention_output_2 = self.attention_2(query=attention_output_1,\n",
    "                                              value=encoder_outputs,\n",
    "                                              key=encoder_outputs,\n",
    "                                              # 将组合掩码传递给第二个注意力层，它将源序列与目标序列相关联\n",
    "                                              attention_mask=padding_mask,)\n",
    "        attention_output_2 = self.layernorm_2(attention_output_1 + attention_output_2)\n",
    "        proj_output = self.dense_proj(attention_output_2)\n",
    "\n",
    "        return self.layernorm_3(attention_output_2 + proj_output)    \n",
    "\n",
    "    def get_causal_attention_mask(self, inputs):\n",
    "        input_shape = tf.shape(inputs)\n",
    "        batch_size, sequence_length = input_shape[0], input_shape[1]\n",
    "        i = tf.range(sequence_length)[:, tf.newaxis]\n",
    "        j = tf.range(sequence_length)\n",
    "\n",
    "    #     生成形状矩阵（sequence_length，sequence_length），一半为 1，另一半为 0\n",
    "        mask = tf.cast(i >= j, dtype=\"int32\")\n",
    "\n",
    "    #     沿批处理轴复制它以获得形状矩阵 (batch_size,序列长度，序列长度）\n",
    "        mask = tf.reshape(mask, (1, input_shape[1], input_shape[1])) \n",
    "        mult = tf.concat([tf.expand_dims(batch_size, -1),\n",
    "                          tf.constant([1, 1], dtype=tf.int32)], axis=0)\n",
    "        return tf.tile(mask, mult)    \n",
    "    \n",
    "    def get_config(self):\n",
    "        config = super().get_config()\n",
    "        config.update({\"embed_dim\": self.embed_dim,\n",
    "                       \"num_heads\": self.num_heads,\n",
    "                       \"dense_dim\": self.dense_dim,})\n",
    "        return config"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "prescription-rehabilitation",
   "metadata": {},
   "source": [
    "**综合：机器翻译的变形金刚**"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "boolean-hampton",
   "metadata": {},
   "source": [
    "端到端 Transformer 是我们将要训练的模型，它将到目前为止的源序列和目标序列映射到未来一步的目标序列。 它直接将我们迄今为止构建的部分组合在一起：PositionalEmbedding 层、TransformerEncoder 和 TransformerDecoder。 请注意，这两个\n",
    "TransformerEncoder TransformerDecoder 和它们是形状不变的，因此您可以将它们中的许多堆叠起来以创建更强大的编码器或解码器。 在我们的示例中，我们将坚持每个实例。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "french-building",
   "metadata": {},
   "source": [
    "> 清单 11.38 端到端变压器"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "id": "olympic-wrong",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "time: 2.28 s (started: 2021-08-05 12:51:38 +08:00)\n"
     ]
    }
   ],
   "source": [
    "embed_dim = 256\n",
    "dense_dim = 2048\n",
    "num_heads = 8\n",
    "\n",
    "encoder_inputs = keras.Input(shape=(None,), dtype=\"int64\", name=\"english\")\n",
    "x = PositionalEmbedding(sequence_length, vocab_size, embed_dim)(encoder_inputs)\n",
    "# 编码源语句\n",
    "encoder_outputs = TransformerEncoder(embed_dim, dense_dim, num_heads)(x) \n",
    "\n",
    "decoder_inputs = keras.Input(shape=(None,), dtype=\"int64\", name=\"spanish\")\n",
    "x = PositionalEmbedding(sequence_length, vocab_size, embed_dim)(decoder_inputs)\n",
    "# 对目标句进行编码，并与编码后的源句组合\n",
    "x = TransformerDecoder(embed_dim, dense_dim, num_heads)(x, encoder_outputs) \n",
    "x = layers.Dropout(0.5)(x)\n",
    "decoder_outputs = layers.Dense(vocab_size, activation=\"softmax\")(x) \n",
    "\n",
    "# 为每个输出位置预测一个单词\n",
    "transformer = keras.Model([encoder_inputs, decoder_inputs], decoder_outputs)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "excessive-jurisdiction",
   "metadata": {},
   "source": [
    "我们现在准备训练我们的模型——我们达到了 67% 的准确率，比基于 GRU 的模型高很多。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "laughing-instrumentation",
   "metadata": {},
   "source": [
    "> 清单 11.39 训练序列到序列转换器"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "id": "motivated-bristol",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/30\n",
      "1302/1302 [==============================] - 165s 120ms/step - loss: 1.0772 - accuracy: 0.6267 - val_loss: 1.0144 - val_accuracy: 0.6330\n",
      "Epoch 2/30\n",
      "1302/1302 [==============================] - 152s 116ms/step - loss: 1.0348 - accuracy: 0.6454 - val_loss: 0.9956 - val_accuracy: 0.6438\n",
      "Epoch 3/30\n",
      "1302/1302 [==============================] - 155s 119ms/step - loss: 1.0077 - accuracy: 0.6595 - val_loss: 0.9856 - val_accuracy: 0.6508\n",
      "Epoch 4/30\n",
      "1302/1302 [==============================] - 156s 120ms/step - loss: 0.9875 - accuracy: 0.6712 - val_loss: 0.9809 - val_accuracy: 0.6535\n",
      "Epoch 5/30\n",
      "1302/1302 [==============================] - 152s 117ms/step - loss: 0.9688 - accuracy: 0.6807 - val_loss: 0.9829 - val_accuracy: 0.6548\n",
      "Epoch 6/30\n",
      "1302/1302 [==============================] - 153s 118ms/step - loss: 0.9530 - accuracy: 0.6894 - val_loss: 0.9798 - val_accuracy: 0.6583\n",
      "Epoch 7/30\n",
      "1302/1302 [==============================] - 154s 118ms/step - loss: 0.9382 - accuracy: 0.6968 - val_loss: 0.9834 - val_accuracy: 0.6605\n",
      "Epoch 8/30\n",
      "1302/1302 [==============================] - 155s 119ms/step - loss: 0.9247 - accuracy: 0.7033 - val_loss: 0.9846 - val_accuracy: 0.6618\n",
      "Epoch 9/30\n",
      "1302/1302 [==============================] - 152s 117ms/step - loss: 0.9123 - accuracy: 0.7092 - val_loss: 0.9872 - val_accuracy: 0.6625\n",
      "Epoch 10/30\n",
      "1302/1302 [==============================] - 154s 118ms/step - loss: 0.9007 - accuracy: 0.7143 - val_loss: 0.9884 - val_accuracy: 0.6653\n",
      "Epoch 11/30\n",
      "1302/1302 [==============================] - 158s 121ms/step - loss: 0.8889 - accuracy: 0.7194 - val_loss: 0.9955 - val_accuracy: 0.6611\n",
      "Epoch 12/30\n",
      "1302/1302 [==============================] - 156s 119ms/step - loss: 0.8781 - accuracy: 0.7237 - val_loss: 0.9907 - val_accuracy: 0.6660\n",
      "Epoch 13/30\n",
      "1302/1302 [==============================] - 154s 119ms/step - loss: 0.8673 - accuracy: 0.7283 - val_loss: 1.0001 - val_accuracy: 0.6642\n",
      "Epoch 14/30\n",
      "1302/1302 [==============================] - 152s 116ms/step - loss: 0.8574 - accuracy: 0.7327 - val_loss: 1.0015 - val_accuracy: 0.6622\n",
      "Epoch 15/30\n",
      "1302/1302 [==============================] - 150s 115ms/step - loss: 0.8478 - accuracy: 0.7361 - val_loss: 1.0042 - val_accuracy: 0.6629\n",
      "Epoch 16/30\n",
      "1302/1302 [==============================] - 155s 119ms/step - loss: 0.8386 - accuracy: 0.7398 - val_loss: 1.0082 - val_accuracy: 0.6646\n",
      "Epoch 17/30\n",
      "1302/1302 [==============================] - 157s 121ms/step - loss: 0.8305 - accuracy: 0.7428 - val_loss: 1.0183 - val_accuracy: 0.6648\n",
      "Epoch 18/30\n",
      "1302/1302 [==============================] - 156s 120ms/step - loss: 0.8215 - accuracy: 0.7461 - val_loss: 1.0117 - val_accuracy: 0.6663\n",
      "Epoch 19/30\n",
      "1302/1302 [==============================] - 155s 119ms/step - loss: 0.8130 - accuracy: 0.7492 - val_loss: 1.0188 - val_accuracy: 0.6636\n",
      "Epoch 20/30\n",
      "1302/1302 [==============================] - 154s 118ms/step - loss: 0.8055 - accuracy: 0.7525 - val_loss: 1.0136 - val_accuracy: 0.6687\n",
      "Epoch 21/30\n",
      "1302/1302 [==============================] - 155s 119ms/step - loss: 0.7975 - accuracy: 0.7553 - val_loss: 1.0243 - val_accuracy: 0.6656\n",
      "Epoch 22/30\n",
      "1302/1302 [==============================] - 156s 120ms/step - loss: 0.7893 - accuracy: 0.7580 - val_loss: 1.0235 - val_accuracy: 0.6664\n",
      "Epoch 23/30\n",
      "1302/1302 [==============================] - 157s 121ms/step - loss: 0.7821 - accuracy: 0.7606 - val_loss: 1.0274 - val_accuracy: 0.6678\n",
      "Epoch 24/30\n",
      " 373/1302 [=======>......................] - ETA: 1:37 - loss: 0.7780 - accuracy: 0.7614"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "IOPub message rate exceeded.\n",
      "The Jupyter server will temporarily stop sending output\n",
      "to the client in order to avoid crashing it.\n",
      "To change this limit, set the config variable\n",
      "`--ServerApp.iopub_msg_rate_limit`.\n",
      "\n",
      "Current values:\n",
      "ServerApp.iopub_msg_rate_limit=1000.0 (msgs/sec)\n",
      "ServerApp.rate_limit_window=3.0 (secs)\n",
      "\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1302/1302 [==============================] - 155s 119ms/step - loss: 0.7689 - accuracy: 0.7659 - val_loss: 1.0446 - val_accuracy: 0.6658\n",
      "Epoch 26/30\n",
      "1302/1302 [==============================] - 153s 117ms/step - loss: 0.7623 - accuracy: 0.7681 - val_loss: 1.0482 - val_accuracy: 0.6678\n",
      "Epoch 27/30\n",
      " 310/1302 [======>.......................] - ETA: 1:44 - loss: 0.7595 - accuracy: 0.7685"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "IOPub message rate exceeded.\n",
      "The Jupyter server will temporarily stop sending output\n",
      "to the client in order to avoid crashing it.\n",
      "To change this limit, set the config variable\n",
      "`--ServerApp.iopub_msg_rate_limit`.\n",
      "\n",
      "Current values:\n",
      "ServerApp.iopub_msg_rate_limit=1000.0 (msgs/sec)\n",
      "ServerApp.rate_limit_window=3.0 (secs)\n",
      "\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1302/1302 [==============================] - 155s 119ms/step - loss: 0.7441 - accuracy: 0.7743 - val_loss: 1.0659 - val_accuracy: 0.6657\n",
      "Epoch 30/30\n",
      "1302/1302 [==============================] - 154s 118ms/step - loss: 0.7385 - accuracy: 0.7759 - val_loss: 1.0671 - val_accuracy: 0.6649\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<tensorflow.python.keras.callbacks.History at 0x7fb88d0ea550>"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "time: 1h 18min 9s (started: 2021-08-05 13:01:48 +08:00)\n"
     ]
    }
   ],
   "source": [
    "transformer.compile(optimizer=\"rmsprop\",\n",
    "                    loss=\"sparse_categorical_crossentropy\",\n",
    "                    metrics=[\"accuracy\"])\n",
    "transformer.fit(train_ds, epochs=30, validation_data=val_ds)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "decreased-brooklyn",
   "metadata": {},
   "source": [
    "最后，让我们尝试使用我们的模型从测试集中翻译从未见过的英语句子。 该设置与我们用于序列到序列 RNN 模型的设置相同。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "furnished-ambassador",
   "metadata": {},
   "source": [
    "> 清单 11.40 使用我们的 Transformer 模型翻译新句子"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "id": "absent-secret",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "-\n",
      "He is just a kid.\n",
      "[start] es sólo un niño [end]\n",
      "-\n",
      "Tom doesn't like cats.\n",
      "[start] a tom no le gustan los gatos [end]\n",
      "-\n",
      "You lean on your parents too much. You must be more independent.\n",
      "[start] te estás de tus padres demasiado padres para tener más más que más te [UNK] [end]\n",
      "-\n",
      "Tom usually stays at five-star hotels.\n",
      "[start] tom normalmente se queda en habitación [UNK] del hotel [end]\n",
      "-\n",
      "She refused his offer.\n",
      "[start] ella negó su los deberes [end]\n",
      "-\n",
      "You'll soon get used to the climate here.\n",
      "[start] pronto te ves [UNK] al clima acá [end]\n",
      "-\n",
      "Although the proposal seemed like a good idea, they refused it.\n",
      "[start] a pesar de que la lo yo parecía una buena idea negó [end]\n",
      "-\n",
      "She felt insecure about her future.\n",
      "[start] ella se sintió a su inglés en su futuro [end]\n",
      "-\n",
      "I must finish it before I go out.\n",
      "[start] debo terminar antes de que yo te vayas [end]\n",
      "-\n",
      "This book is to me what the Bible is to you.\n",
      "[start] este libro me tiene hasta la culpa de la culpa [end]\n",
      "-\n",
      "It is possible that you have already read this book.\n",
      "[start] ya es posible que ya te [UNK] este libro [end]\n",
      "-\n",
      "Mind your manners.\n",
      "[start] [UNK] tus lo se [UNK] [end]\n",
      "-\n",
      "Is this a pen or a pencil?\n",
      "[start] este es un bolígrafo o un diario [end]\n",
      "-\n",
      "Tom did it by himself.\n",
      "[start] tom lo hizo por sí solo [end]\n",
      "-\n",
      "I wake him at six every morning.\n",
      "[start] lo conocí a las 6 cada mañana [end]\n",
      "-\n",
      "We all stood up at once.\n",
      "[start] todos nosotros [UNK] al día [end]\n",
      "-\n",
      "He is used to that type of situation.\n",
      "[start] Él está acostumbrado a ese tipo de situación [end]\n",
      "-\n",
      "I thought he would come.\n",
      "[start] pensé que él vendría [end]\n",
      "-\n",
      "I can't breathe.\n",
      "[start] no puedo la alarma [end]\n",
      "-\n",
      "She likes to read books.\n",
      "[start] a ella le gusta leer libros [end]\n",
      "time: 27.9 s (started: 2021-08-05 22:25:52 +08:00)\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "\n",
    "spa_vocab = target_vectorization.get_vocabulary()\n",
    "spa_index_lookup = dict(zip(range(len(spa_vocab)), spa_vocab))\n",
    "max_decoded_sentence_length = 20\n",
    "    \n",
    "def decode_sequence(input_sentence):\n",
    "    tokenized_input_sentence = source_vectorization([input_sentence])\n",
    "    decoded_sentence = \"[start]\"\n",
    "    for i in range(max_decoded_sentence_length):\n",
    "        tokenized_target_sentence = target_vectorization(\n",
    "        [decoded_sentence])[:, :-1]\n",
    "#         采样下一个令牌\n",
    "        predictions = transformer([tokenized_input_sentence, tokenized_target_sentence]) \n",
    "        sampled_token_index = np.argmax(predictions[0, i, :])\n",
    "        # 将下一个标记预测转换为字符串并将其附加到生成的句子中\n",
    "        sampled_token = spa_index_lookup[sampled_token_index]\n",
    "        decoded_sentence += \" \" + sampled_token\n",
    "        # 退出条件\n",
    "        if sampled_token == \"[end]\":\n",
    "            break\n",
    "    return decoded_sentence\n",
    "\n",
    "test_eng_texts = [pair[0] for pair in test_pairs]\n",
    "for _ in range(20):\n",
    "    input_sentence = random.choice(test_eng_texts)\n",
    "    print(\"-\")\n",
    "    print(input_sentence)\n",
    "    print(decode_sequence(input_sentence))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "known-chemistry",
   "metadata": {},
   "source": [
    "主观上，Transformer 的性能似乎明显优于基于 GRU 的翻译模型。 它仍然是一个玩具模型，但它是一个更好的玩具模型。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "exact-revolution",
   "metadata": {},
   "source": [
    "> 清单 11.41 Transformer 翻译模型的一些示例结果"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "english-dating",
   "metadata": {},
   "outputs": [],
   "source": [
    "This is a song I learned when I was a kid.\n",
    "# 虽然源句子没有性别化，但这个翻译假设是男性说话者。 \n",
    "# 请记住，翻译模型通常会对其输入数据做出无根据的假设，从而导致算法偏差。 \n",
    "# 在最坏的情况下，模型可能会产生与当前正在处理的数据无关的记忆信息的幻觉。\n",
    "[start] esta es una canción que aprendí cuando era chico [end] \n",
    "-\n",
    "She can play the piano.\n",
    "[start] ella puede tocar piano [end]\n",
    "-\n",
    "I'm not who you think I am.\n",
    "[start] no soy la persona que tú creo que soy [end]\n",
    "-\n",
    "It may have rained a little last night.\n",
    "[start] puede que llueve un poco el pasado [end]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "adequate-navigator",
   "metadata": {},
   "source": [
    "关于自然语言处理的本章到此结束——您刚刚从最基本的知识发展到了一个可以将英语翻译成西班牙语的成熟 Transformer。 教机器理解语言是您可以添加到收藏中的最新超能力。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "accepted-smooth",
   "metadata": {},
   "source": [
    "## 章节总结"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "beneficial-organizer",
   "metadata": {},
   "source": [
    "* 有两种 NLP 模型：词袋模型，处理词集或 n-gram 而不考虑它们的顺序，以及序列模型，处理词序。词袋模型由密集层组成，而序列模型可以是 RNN、1D convnet 或 Transformer。\n",
    "* 在文本分类方面，训练数据中的样本数与每个样本的平均词数之间的比率可以帮助您确定应该使用词袋模型还是序列模型。\n",
    "* 词嵌入是向量空间，其中词之间的语义关系被建模为表示这些词的向量之间的距离关系。\n",
    "* 序列到序列学习是一个通用的、强大的学习框架，可用于解决许多 NLP 问题，包括机器翻译。序列到序列模型由处理源序列的编码器和试图通过在编码器处理的源序列的帮助下查看过去的标记来预测目标序列中未来标记的解码器组成。\n",
    "* 神经注意力是一种创建上下文感知词表示的方法。它是 Transformer 架构的基础。\n",
    "* Transformer 架构由一个 TransformerEncoder 和一个 TransformerDecoder 组成，在序列到序列任务上产生了出色的结果。前半部分 TransformerEncoder 也可用于文本分类或任何类型的单输入 NLP 任务。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "chubby-cleanup",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python [conda env:tensorflow]",
   "language": "python",
   "name": "conda-env-tensorflow-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
